To better understand patient satisfaction and perceived engagement with traditional hospital-based communication and to elicit patient preferences for health information technologies that would lead to improved satisfaction and engagement.

We performed a mixed-methods study involving qualitative interviews followed by a survey of hospitalized patients and their family members at a single large academic medical center. 

We conducted semi-structured interviews with 41 patients and surveyed 267 patients or family members to elicit their perspectives on satisfaction with traditional hospital communication methods, information needed to more fully engage in the patients' medical care, and potential solutions for improved hospital-based communication. 

Qualitative interviews revealed patients’ and family members’ dissatisfaction with current hospital-based communication methods. They would prefer more information, in more flexible forms, with real-time digital access and the ability to share within their social and healthcare networks. Quantitative results from the survey supported these premises, with at least the majority of the 267 patients surveyed agreeing across each survey question. Furthermore, participants identified a “communications point person” as the individual who organizes, understands, and communicates about the patient’s care, who was often a family member not available at the bedside during daily rounds. Potential solutions included improved transparency about hospital processes, creating systems that allow patients and family to help coordinate and double-check their own health-related communications, and delivering hospital-based communications through digital media. 

These study findings provide empiric evidence to hospital decision-makers regarding patient and family preferences for 21st-century hospital-based communication systems. 

Patients and family members perceived mismatches between current hospital communications and their expectations.

Hospitalized patients and family members wanted to be more engaged in the patients' healthcare and perceived that enhanced, transparent, and easy-to-access information would help them accomplish this goal.

Hospital policy-makers should consider communication platforms that better meet the needs of patients and family members.

With the goal of engaging patients and family members to more fully participate in their own healthcare and improve health-related behaviors, Meaningful Use regulations encourage enhanced communication with patients and family via health information technologies, such as patient portals.

 Accordingly, outpatient-based patient portals have been shown to be effective in engaging patients through transparent access to medical records.

Despite well-documented success in outpatient settings, few study findings show successful large-scale implementation of hospital-based patient portals.

 Although the reasons for generalized low engagement by hospitalized patients and family members are not fully understood and may be multifactorial, some authors have suggested that hospital-based patient portals have not been created to optimize user functionality.

 Gaps remain in knowledge regarding how to effectively engage patients and family members, the level of information that would improve patient satisfaction and outcomes, and specific health information technology platforms that are most effective in the hospital setting. 

Our study aimed to better understand 1) patient satisfaction and perceived engagement with traditional hospital-based communications and 2) patient preferences for transparent data and health information technologies that would lead to improved patient satisfaction and enhanced engagement in the hospital setting.

We performed a mixed-methods study of hospitalized patients and their family members at a single large academic medical center with a daily census of approximately 640 inpatients. First, we conducted a qualitative analysis of 41 participant interviews to explore the range of ideas perceived by patients and family members. Second, we employed random sampling and surveyed 267 participants to quantify the proportion of hospitalized patients and family members who agreed with each of the major ideas developed during the qualitative analysis. All research activities were approved by the Albany Medical College institutional review board. 

Our research team first performed qualitative interviews with a convenience sample of 41 hospitalized patients and their family members on the medical nursing units from November 2014 to February 2015. Participants were recruited until the study investigators, who met periodically during the data collection period, determined that saturation in the answers to the open-ended questions had been reached and no new concepts were discovered. The qualitative interviews were done using a semi-structured format, with responses recorded in real time. Individual concepts were categorized into discrete themes, and a qualitative analysis was performed. Three study investigators participated in coding of themes and subthemes, and a fourth investigator reviewed coded themes for inconsistencies (further detail of the qualitative analysis can be found in the 

Based on the results of this qualitative analysis, we developed a quantitative survey for patients (or their family members) that consisted of questions related to 3 distinct overarching themes identified in the initial qualitative interviews: 1) satisfaction with traditional hospital communication methods, 2) perspectives on information needed to more fully engage in their medical care, and 3) potential solutions for improved hospital-based communication. 

The quantitative survey instrument was refined in an iterative fashion using the input of an additional 93 patients and family members. This included cognitively testing the survey questions on sequential potential participants until the research team was confident that there was appropriate content, comprehension, and quality (61 participants)

 and pilot-testing the entire survey to ensure clarity and prevent survey fatigue (32 participants). 

Patient Selection for Quantitative Survey

Patients were randomly selected (using a random number generator) daily from the medical nursing units between 9 am and 5 pm from January 2016 to May 2016. Patients were included if they were 18 years or older, admitted more than 48 hours prior to the time of screening, and English-speaking. Only 1 study participant (either the patient or a family member) was allowed per hospitalized patient. Patients who were deemed to not have capacity to answer questions (based on nursing assessment) and who did not have a family member at the bedside were excluded. When a family member was available, patients were given the option to complete the survey themselves or to have a family member complete it. Verbal consent was obtained by a trained research assistant. 

Data Collection and Analysis of Quantitative Survey

The survey was administered on an iPad, with the research assistant available to assist as needed. Age was sorted, in part, by generational labeling: under 27 years (Millennial), 28 to 51 years (Generation X), 52 to 70 years (Baby Boomer), 71 to 88 years, and over 88 years.

Three survey questions had answers of “Yes,” “No,” or “This does not concern me.” Twenty-three survey questions had standardized “Agree” answers on a 4-point Likert scale, with the additional option of “This does not concern me.” Through the cognitive testing of the survey questions mentioned above, we developed the option of “This does not concern me” for situations in which either the participant did not have an opinion or the question raised a scenario that was not applicable to their medical care. When appropriate, to dichotomize results, “Strongly agree” and “Agree” were grouped together to indicate the response as “Agree.” We analyzed survey responses by respondent characteristics using a χ

 analysis. Data analysis was performed using Stata version 14.0 (StataCorp LLC; College Station, Texas).

Participant Characteristics and Patient Demographics

For the qualitative interviews, there were 4 patients aged 18 or 19 years, 5 patients aged 20 to 39 years, 10 patients aged 40 to 59 years, 15 patients aged 60 to 79 years, and 6 patients 80 years or older (categorical age was missing for 1 patient). Forty-five percent of the patients were female (gender status was missing for 1 patient). 

The final sample of 267 survey participants included 216 (81%) patients and 51 (19%) family members. 

 depicts the flow of participant eligibility assessment, recruitment, enrollment, and survey participation. 

 lists participant characteristics and patient demographics.

Satisfaction With Traditional Hospital Communication Methods

The initial qualitative analysis revealed that patients and family members were dissatisfied with current hospital communication methods. In particular, respondents felt that important information about their healthcare was not being communicated to them and expressed frustration with face-to-face interactions as the sole mode of communication. Participants felt this process lacked effectiveness both in understanding the complexity of their medical care and in relaying important health information to family members and the patient’s health support network. 

Additional quantitative analysis supported these findings. Although the majority of patients agreed that they knew why they were taking each medication in the hospital and why each test was being done, 63% felt that there was more information available than was being communicated by their care team, and 71% reported that they sometimes forgot information the doctor told them. Furthermore, only 38% of participants perceived that it was easy to contact their doctor outside of their daily interactions and less than half (46%) that it was easy to coordinate with their entire healthcare team and health support network regarding their discharge planning (

Defining the Communications Point Person 

A second overarching theme identified in the qualitative analysis was the importance of an individual who served as the communications hub to relay the patient’s health information to family, the patient’s healthcare providers, and others within the patient’s health support network. Patients and family identified circumstances in which a family member was not necessarily the medical decision-maker for the patient, but was responsible for coordinating among all involved parties. We have termed this the communications point person (CPP), defined as the person who organizes, understands, and communicates about the patient’s care. 

In support of this concept, the quantitative survey found that 182 (68%) participants identified someone other than the patient (family member, friend, or other) as the CPP, whereas 85 (32%) identified the patient as the CPP (

). Of those who identified someone other than the patient, respondents identified the CPP as a spouse (38%), daughter (32%), son (14%), parent (11%), sister (8%), partner (6%), brother (2%), or another person (defined as friend, cousin, nephew, niece, or grandparent) (15%). (Participants were able to identify more than 1 CPP, producing a total greater than 100%.)

Of the 182 participants for whom the CPP was someone other than the patient, information was elicited on CPP availability for face-to-face in-hospital communication. Responses indicated CPP availability during the morning (39%), afternoon (45%), evening (41%), or night (6%). (Participants were able to identify more than 1 time of day, for a total greater than 100%.) Additionally, 30% of respondents reported that the CPP’s availability varied by day and 7% of respondents reported that their CPP was unable to be at the hospital at all. 

Information Required by Patients/Family Members to Be Engaged in Their Medical Care

During our qualitative interviews, we found that patients and family members wanted to be more engaged in their healthcare. They relayed that they did not have access to all of the information they needed (eg, test results, hospital medications, etc) and that they wanted more specific and transparent information in real time in the hospital setting. The quantitative survey results supported these findings. Ninety percent of participants agreed that it would be helpful to have access to all of their health information (including test results and medications) while they were in the hospital, and 79% felt they could be more involved in their medical care if they had access to all of their health information. 

Furthermore, most participants (83%) wanted unfettered real-time access to their medical records. This was true even when we challenged participants to imagine scenarios in which their doctor had not yet had a chance to review the test results (74% of the original survey cohort still wanted access to their medical records) and when the test results revealed a life-altering finding (57% of the original cohort still wanted access).

Potential Solutions for Improved Hospital-Based Communications

We identified possible hospital-based communications solutions during qualitative interviews. Patients and family wanted 1) to be more prepared for their discussions with their healthcare providers; 2) transparency about in-hospital processes, including communications among hospital providers and status updates on the care plan; 3) a solution that assists in coordination of care among healthcare providers, family members, and others in their health support network; and 4) consumer-friendly digital access to their hospital information (eg, access to an easy-to-use phone, tablet, or computer system that enables them to interact with their medical records and hospital providers). 

Further quantitative analyses revealed that 78% of participants agreed that face-to-face interactions would be more meaningful if they had access to their test results, along with educational materials, prior to speaking to the doctor; 90% of participants wanted to verify that communication took place among their hospital providers; and 87% wanted to know when the doctor reviewed their test results. Moreover, 80% of participants thought it would be helpful to have access to a digital system that records activities and results during their hospitalization, and 85% of participants thought it would be helpful to securely share some or all of their health information while in the hospital.

Comparison of Survey Results by Participant Characteristics and Patient Demographics

For 11 of the 25 questions, family member respondents were statistically significantly (

 <.05) more likely than patient respondents to choose answers that agreed with more transparent, user-friendly communications that would produce more patient and family engagement (Figure 2). Despite this, family members were statistically significantly (

 <.05) more likely to report that they knew why each test was being done. 

Patient respondents younger than 52 years were statistically significantly (

 <.05) more likely than those 52 years or older to agree that they wanted the option to see all of their test results, wanted to see their health information in real time, and believed it would be helpful to have transparent access on their phone, tablet, or computer to “everything” regarding their medical care. There were no statistically significant differences (all 

 >.05) in answers given by frequently hospitalized patients, defined as patients with 3 or more admissions over the past 3 years, versus other hospitalized patients.

Although communication systems in the outpatient setting are adapting to newer technologies, hospitals have been slow to complement traditional face-to-face interactions with updated communication modalities. Our study found that hospitalized patients and their family members perceived themselves to be disempowered, as they remained dissatisfied with the extent of information provided, frequency of communication, level of access to data, and current communication modalities. We used the term communications mismatches to define and label the disparities between traditional in-hospital communications and the expectations that patients and families have regarding today’s social structures and the digital network environment. 

As an example, when interviewing patients and family, we discovered that in many circumstances, there was 1 family member who took on the role of organizing, understanding, and communicating about the patient’s care, who we defined as the CPP. Whereas the patient’s healthcare proxy (proxy medical decision-maker) and surrogates/care partners (individuals allowed access to the patient’s medical records and communication with healthcare providers) are technically or legally defined terms,

 the CPP is a patient-derived concept of the key communications leader in each patient’s health support network. 

Not surprisingly, our quantitative survey revealed a communications mismatch between the CPP’s availability and typical bedside rounds, as most CPPs were not available during the morning, which is when physicians typically perform face-to-face rounds. These empiric findings quantify, and emphasize, the need to identify the CPP for each patient and create more flexible communications systems to empower CPPs in their role within each patient’s social network. Although the concept of identifying and digitally connecting with care partners has been explored in the outpatient setting,

 there is limited discussion in the medical literature on this topic in the hospital setting. 

Active engagement of patients and family to participate in the maintenance of their own health has gained importance on the national healthcare agenda.

 Available research in the outpatient setting suggests that patient portals empower patients to participate in their own care

 and shared decision making has become an important feature of high-quality patient care.

 At least 1 prior study has broached this topic in the hospital setting and found that patients wanted more access and detail in their medical records, although these findings came mostly from qualitative interviews.

 Our study also found that the majority of hospitalized patients and family wanted to be more engaged in their hospital care. Patients and family members believed that more transparent and user-friendly access to their medical records would empower them to track their medical progress, better understand and plan for the discharge process, and identify errors in their medical records. Moreover, most participants believed that reviewing their results before their provider interactions would ultimately prepare them for and augment these important discussions and that access to their medical records would enable them to serve as their own quality-assurance and patient-safety advocate. Furthermore, we found that most participants wanted to see their test results in real time, even when their providers had not yet viewed the test results or in situations that involved life-changing news. 

Finally, we explored patient- and family-driven solutions to meet communication needs and wants in the hospital setting. Patients and family members wanted hospitals to 1) enhance the amount, level, and timing of information and access to medical data via consumer-friendly digital platforms; 2) actively identify the CPP for each patient and ensure communication with the CPP through appropriate timing and modalities; 3) allow the opportunity for medical information to be easily and securely shared with family, healthcare providers, and others in the health support network; and 4) provide a forum for verification and updated status on steps and processes that are part of the hospital healthcare plan. These strategies may be particularly important for younger patients and family, as Millennial and Generation X respondents wanted more information and more real-time access on digital platforms compared with participants classified as Baby Boomer generation and older.

Articles

WEB EXCLUSIVE

This mixed-methods study evaluated hospitalized patients’ and family members’ perceived communications mismatches and their calls for transparent real-time information and potential 21st-century solutions.

Hospitalized Patients' and Family Members' Preferences for Real-Time, Transparent Access to Their Hospital Records

 We examined racial/ethnic variation in the devices used by patients to access medical records through an online patient portal.

 Retrospective, cross-sectional analysis.

 Using data from 318,700 adults enrolled in an integrated delivery system between December 2012 and November 2013, we examined: 1) online patient portal use that directly engages the electronic health record and 2) portal use over desktops/laptops only, mobile devices only, or both device types. The primary covariate was race/ethnicity (non-Hispanic white, black, Hispanic, and Asian). Other covariates included age, sex, primary language, and neighborhood-level income and education. Portal use and devices used were assessed with multiple and multinomial logistic models, respectively.

 From December 2012 to November 2013, 56% of enrollees used the patient portal. Of these portal users, 62% used desktops/laptops only, 6% used mobile devices only, and 32% used both desktops/laptops and mobile devices. Black, Hispanic, and Asian enrollees had significantly lower odds of portal use than whites. Black and Hispanic portal users also were significantly more likely to use mobile devices only (relative risk ratio, 1.73 and 1.44, respectively) and both device types (1.21 and 1.07, respectively) than desktops/laptops only compared with whites.

 Although racial/ethnic minority enrollees were less likely to access the online patient portal overall, a greater proportion of black and Hispanic users accessed the patient portal with mobile devices than did non-Hispanic white users. The rapid spread of mobile devices among racial/ethnic minorities may help reduce variation in online patient portal use. Mobile device use may represent an opportunity for healthcare organizations to further engage black and Hispanic enrollees in online patient portal use.

The increasing availability of mobile devices and applications has the potential to extend patient engagement through the use of patient portals. This study examined racial/ethnic variation in the devices used (desktop/laptop computer, mobile device, or both device types) by patients to access medical records through an online patient portal. 

Black and Hispanic online patient portal users were significantly more likely to use mobile devices exclusively and both device types than desktop/laptop computers exclusively compared with non-Hispanic white portal users. 

Mobile device use may represent an opportunity for healthcare organizations to further engage black and Hispanic enrollees in online patient portal use.

Online patient portals have the potential to be important health communication and care coordination tools for patients and providers and to improve access to healthcare for patients.

 Healthcare systems and clinics use patient portals to fulfill key metrics of federal Meaningful Use requirements, including direct patient engagement; this occurs when patients view, download, and transmit health information online and use secure messaging to communicate with providers.

 In 2010, the use of secure online patient portals by adults nationally doubled from 2 years before to 7%,

 while integrated and academic healthcare systems reported use by more than 40% of patients.

 Patient portals provide secure access to and interaction with healthcare-related services and information in electronic health records (EHRs).

 Patient portal functions may include secure messaging with providers, requesting medication refills and appointments, and viewing laboratory results and other portions of the EHR. Use of portals has been linked to improved care quality, patient satisfaction, and health outcomes.

Portal enrollment and use are generally lower among racial/ethnic minority patient populations.

 Specifically, black, Hispanic, and Asian patients have been found to have lower enrollment and use of patient portals than non-Hispanic white patients, even after adjusting for internet access and use.

 However, new technologies, such as smartphones and other mobile devices, may help attenuate differences in patient portal use among racial/ethnic minority patients. Mobile device browsers can be used to access patient portal websites, and healthcare organizations are implementing mobile applications with similar sets of services.

 Blacks and English-speaking Hispanics are more likely to own a smartphone and use a wider range of its applications (eg, internet, email) than whites.

 Compared with whites, blacks and English-speaking Hispanics also are more likely to rely on their smartphones for access to the internet.

 Although racial/ethnic minorities are more likely to own and rely on mobile devices, the current extent of access to patient portals with mobile devices is unknown.

We examined the relationship of race/ethnicity to patient portal use and devices used to access the portal among enrollees in an integrated healthcare delivery system. We focused on patient portal functions associated with the Meaningful Use criteria, specifically measuring patient healthcare management tasks that engage the healthcare provider and healthcare system. Similar to previous findings, we hypothesized that non-Hispanic whites would be more likely to use the patient portal than racial/ethnic minority enrollees. Given the high rates of smartphone use among racial/ethnic minorities, we further hypothesized that black and Hispanic enrollees would have a higher proportion of patient portal use via mobile devices (compared with desktop/laptop computers) than non-Hispanic white enrollees.

Setting, Data Sources, and Study Participants

This study was conducted in Group Health Cooperative (now Kaiser Permanente Washington), a mixed-model healthcare delivery system in Washington state. Approximately 400,000 members receive their care through Group Health’s integrated delivery system, which includes 25 Group Health-owned facilities and approximately 1000 Group Health physicians. The online patient portal has been available on the patient website (ghc.org during the time of the study, currently kp.org) since 2003 and on a mobile application (for both iPhone and Android) since 2011. Enrollees are informed of the website and portal functions upon enrollment. Mobile application use is encouraged via several marketing methods, including the patient website and screensavers throughout clinics. To access the patient portal, patients register online on the patient website (through a desktop/laptop or mobile Web browser) and verify their identity at a clinic or register in person at any clinic.

 Once registered, patients can access a number of features, including portal functions. Access is available to all patients in the integrated delivery system and is available only in English. 

This cross-sectional study drew from 5 enrollee data sources: administrative data for individual demographic information, Web server logs to identify devices used to access functions, EHR records to identify functions accessed, online prescription refill data to identify medication refill requests, and 2010 US Census data for Census block-level demographic information.

Study enrollees were adults (≥18 years) who were enrolled in Group Health’s integrated delivery system for any length of time from December 2012 to November 2013 and who had a primary care provider in a clinic owned and operated by Group Health. Adults receiving care in Group Health’s contracted network were excluded because some portal functions were not available to them.

The Group Health Research Institute’s Institutional Review Board approved this study. 

Our primary outcomes of interest were 1) portal use among study enrollees and 2) devices used to access the patient portal among portal users.

We defined portal use as having used any of 8 eligible portal functions that directly engaged patients with the EHR on at least 2 days within the study period. The 8 portal functions align with federal Meaningful Use requirements: secure messaging with providers, requesting medication refills and appointments, and viewing after-visit summaries, medical test results, medical conditions, allergies, and immunizations.

 We defined portal use as 2 uses on 2 days to identify meaningful engagement with the EHR (rather than a single log-on) and to allow for multiple devices to be used for accessing functions.

Devices used to access the patient portal were identified through server logs.

 We parsed HTTP request strings from Web server log data to identify specific devices (eg, smartphones, tablets, desktops/laptops) and mode of accessing the data (eg, Web browser, mobile application). Log data were matched with EHR records within a time window. All 8 portal functions were accessible through all devices and modes of access.

We categorized devices used as desktop computer only, mobile devices only, and both device types. “Desktop computer only” captured both desktop and laptop computer use; our methodology could not distinguish between these devices and the similar user interface (eg, a physical keyboard) of laptop and desktop computers warranted their combination. “Mobile devices only” indicated use of smartphones and tablets to access the portal through the mobile application or Web browser. “Both device types” indicated use of desktop/laptop computers and mobile devices.

Our independent variables were race/ethnicity, age, gender, primary language, and neighborhood-level education and income. 

We obtained race/ethnicity, gender, age (18-34, 35-49, 50-64, or ≥65 years), and primary language (English or any other language) from administrative data. These variables were collected mostly by administrative staff at enrollee registration in the plan. Self-reported race/ethnicity was our primary independent variable and was categorized as Hispanic, non-Hispanic white, non-Hispanic black, non-Hispanic Asian, and non-Hispanic other. “Other” included enrollees who self-identified as Native American, Hawaiian/Pacific Islander, or with 2 or more races.

Because enrollee education and income had not been collected at registration, we linked enrollees’ home addresses to Census block group-level data and determined the percentage of individuals in the Census block group who completed high school and the group’s median household income.

For bivariate analyses, we examined descriptive characteristics by race/ethnicity using χ

 tests and examined pairwise comparisons for device use and portal use by race/ethnicity (compared with white) using 2-sample tests of proportions. We used multiple logistic regression to determine independent associations between race/ethnicity and portal use among all study enrollees. We used multinomial logistic regression to determine the relationship between race/ethnicity and devices used to access the patient portal among portal users. We report these findings as relative risk ratios; values greater than 1 suggest greater relative risk than the reference group, whereas values less than 1 suggest lower relative risk. Sensitivity analyses including the “missing” race/ethnicity and primary language category from regression models were conducted to assess the impact of missing information on results. We present descriptive information for enrollees missing race/ethnicity and primary language, but excluded these categories from the regression models because results with these categories were similar in significance and direction.

All analyses were conducted using Stata version 13.0 (StataCorp LLC; College Station, Texas). 

Among 318,700 Group Health enrollees, 68% were white; 4%, black; 8%, Asian; and 5%, Hispanic (

). Enrollees varied significantly across demographic characteristics. Twenty-six percent of white enrollees were 65 years or older compared with 11% to 14% of racial/ethnic minority enrollees. Asians and Hispanics had significantly higher proportions of enrollees with a primary language other than English (17% and 7%, respectively). Blacks had the lowest proportion living in high educational attainment areas (89%) and the highest proportion in the lowest median household income areas (43%). Mean months of enrollment during the study period did not vary by race/ethnicity (data not shown). 

Race/ethnicity and primary language were missing for 10% and 9%, respectively, of study enrollees. Race/ethnicity data also were most likely to be missing among younger male enrollees. 

Unadjusted Patient Portal Use and Devices Used to Access the Patient Portal

Among the study sample, 74% registered for the patient portal and 56% used the portal during the study period (

). Portal registration and use was highest among white enrollees (81% and 64%, respectively) and lowest among black enrollees (64% and 44%). The majority of portal users accessed the following: medical test results (87%), secure messaging (68%), and medication refill requests (54%). Compared with whites, racial/ethnic minority users were less likely to use secure messaging and submit refill requests, but similarly or more likely to view medical conditions, immunizations, and allergies. Compared with enrollees whose primary language was English, enrollees with another primary language were significantly less likely to use the patient portal. These portal users were less likely to use secure messaging, request appointments and medication refills, and view after-visit summaries (

The devices used to access the portal varied by race/ethnicity (

). Sixty-two percent used desktop computers only, 6% used mobile devices only, and 32% used both device types. White portal users had the highest proportion using only desktop computers (63%), whereas Black and Hispanic portal users had a higher proportion using only mobile devices (10% and 9%, respectively) compared with lower mobile-only use among white portal users (5%). Using both device types to access the portal was higher among all racial/ethnic minority portal users. 

Variations in Portal Use by Race/Ethnicity and Primary Language

After adjustment for age, gender, primary language, neighborhood-level education, and neighborhood-level income, differences in portal use remained significant for all racial/ethnic minority enrollees compared with white enrollees (

). Black, Asian, and Hispanic enrollees had significantly lower odds of portal use than white enrollees (odds ratios [ORs]: 0.49, 0.70, and 0.70, respectively). Additionally, enrollees whose primary language was not English had significantly lower odds of portal use than those whose primary language was English (OR, 0.24). 

Variations in Devices Used to Access the Portal

Device used to access the patient portal varied among all portal users (

). In the unadjusted model, all racial/ethnic minority portal users were significantly more likely to use mobile devices and both device types than desktops only compared with white portal users. After adjustment, compared with white portal users, black and Hispanic portal users were more likely to use mobile devices only than desktops only to access the portal (relative risk ratio [RRR], 1.73 and 1.44, respectively; both 

 <.05) and more likely to use both device types (RRR, 1.21 and 1.07; both 

 <.05). After adjustment, Asian portal users were less likely to use both device types than desktops only (RRR, 0.93; 

 <.05) compared with whites. Asian portal users also were no longer significantly different from white users in their use of mobile devices only. Primary language was not significantly associated with device used. 

We observed important racial and ethnic variations in use of the online patient portal and in the proportion of devices enrollees used to access the portal in an integrated healthcare system. Although most patient portal users continued to use solely desktop or laptop computers to access the portal, 10% of black and 9% of Hispanic enrollees used mobile devices exclusively to access their patient portals during the study year. Black and Hispanic portal users were more likely than white portal users to use mobile devices only or both device types to access their portals. Although overall portal use was high among all enrollees, racial/ethnic minority enrollees remained less likely to use the portal compared with white enrollees. Almost two-thirds of white enrollees used the portal, yet approximately half or less of racial/ethnic minority enrollees used the portal.

Our study is one of the first to examine variations in the devices enrollees use to access the patient portal. The higher mobile-only portal access among black and Hispanic users may be linked to greater smartphone dependency, preferences and attitudes toward mobile device use (not explained by income or education), and affordability of different devices.

 In addition to higher ownership rates of mobile devices and a wider usage of mobile data applications, 12% of blacks and 13% of English-speaking Hispanics are smartphone dependent (ie, lacking alternatives to their smartphone for internet access) compared with 4% of whites.

 This suggests that a proportion of our mobile-only population may not otherwise have the ability to access their patient portal. Longitudinal studies are needed to assess whether the use of mobile devices and applications to access patient portals are similar across racial/ethnic minorities or if higher uptake of mobile devices among racial/ethnic minorities may attenuate racial/ethnic differences in portal use over time. Surprisingly, Asian portal users had a similar likelihood of exclusive mobile device use but a lower likelihood of using both device types compared with white users. This finding merits further research, as smartphone ownership and home internet use are highest among Asians.

 With 62% of smartphone owners using their phones to access health information,

 this cross-sectional analysis suggests that the mobile-only user population represents an opportunity for healthcare organizations to engage black and Hispanic patients in portal use. However, it is important to recognize that individuals and households who rely on mobile devices as their only computer are often younger, lower-income, racial/ethnic minority populations who are more vulnerable to financial and technical constraints (eg, cost of maintaining a cell phone plan, data allowances).

 Further research is needed to better understand the barriers that these and other vulnerable populations (eg, elderly patients or those with lower educational attainment) face in accessing patient portals.

 patient portal use was lower among racial/ethnic minority patients. Although black and Asian enrollees in our healthcare system were previously found to have lower odds of portal use,

 the similarity in observed difference in portal use in Asians and Hispanics compared with whites (both ORs, 0.70) was unexpected. Nationally, Asians have the highest internet accessibility and use of all races/ethnicities,

 and previous studies have observed either no significant difference or more portal use in Asians compared with whites.

 Our finding may be driven by factors not captured in our data, and further investigation should explore how portal use may vary across Asian subgroups or factors such as patient portal awareness among Asian enrollees. Lower portal use among racial/ethnic minorities in our sample appeared to be due to lower use of active functions (eg, secure messaging and medical refills), rather than passive functions (eg, viewing medical tests).

 Addressing the use of more complex functions may help increase overall use among racial/ethnic minority patients. Similar to findings of previous studies,

 portal use was lower among enrollees whose primary language was not English. However, primary language was not associated with device used. This suggests that language is an important barrier to portal use, but not to how patients access the portal. Differences in portal use may arise from difficulties in navigating an English-language patient portal, which contains the dual barriers of language literacy and health literacy. Additional reasons, such as computer literacy and communication preferences, should also be explored. Lower portal use among racial/ethnic minority populations underscores existing concerns about the digital divide and its potential to amplify healthcare-related disparities among vulnerable racial/ethnic minority enrollees.

 Differences in portal use by race/ethnicity and primary language suggest that targeted promotion and development of the patient portal may be needed.

There are several important limitations to note. First, we were unable to determine many enrollee characteristics, such as literacy (including reading/writing, computer, and health), internet accessibility, mobile device use preferences (eg, accessing the internet and potentially sensitive information), and individual socioeconomic factors. Although we were unable to control for internet accessibility, particularly mobile device ownership, previous studies measuring any internet use found racial/ethnic disparities to persist in portal use.

 Third, we observed higher levels of missing information for language and race/ethnicity; however, these data were self-reported in Group Health datasets according to organizational standards. Prior studies have also reported high rates of missing information among these variables.

 Notably, Meaningful Use Stage 1 requires healthcare organizations to have these fields populated for more than 50% of patients, as both fields need to be better populated to measure possible enrollee disparities.

 Fourth, we measured portal use as using eligible portal functions on at least 2 days. This may have biased the portal user sample toward patients with chronic medical conditions and away from younger, healthier patients who seldom use healthcare services. However, we believe this measure better captures meaningful engagement in care compared with other metrics of website use and allows all categories of devices used (ie, both device types) to be endorsed equally. Fifth, most Web server log entries did not have identifiers to link enrollees to devices, and we matched patient portal records to identifiable log entries using the function’s timestamp; we validated this approach through use and data capture in the EHR and portal test environment.

 Although this approach may have resulted in some underestimation of overall portal use, we do not believe this method caused any systematic bias by race/ethnicity. Finally, Group Health is an integrated delivery system and its racial/ethnic composition differs from that of the general population; these findings may not be generalizable to other healthcare systems. However, this study more accurately captures devices used to access the portal by collecting electronic data, rather than relying on self-reported use data; this removes the information bias that is a common concern in patient behavior studies.

Mobile devices are playing a large role in healthcare, from providing on-demand health information to helping patients and providers monitor and manage chronic conditions. However, researchers and healthcare organizations need to ensure that our increasing reliance on mobile technology does not exacerbate disparities in care and access among vulnerable populations.

 In addition to reconfirming racial/ethnic differences in online patient portal use, we found black and Hispanic portal users to be more likely to use mobile devices to access the patient portal. Mobile devices present an opportunity for healthcare organizations to increase their patient engagement among black and Hispanic enrollees. The accessibility and functionality of patient portals through mobile applications and mobile browsers may help reduce differences in patient portal use.

 RTI International (EC), Waltham, MA; Medical Directorate, University Hospitals of Geneva (KB), Geneva, Switzerland; UCSF Division of General Internal Medicine and Center for Vulnerable Populations at Zuckerberg San Francisco General Hospital (CRL), San Francisco, CA; Kaiser Permanente Washington Health Research Institute (LJ, JDR), Seattle, WA.

 This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Dr Chang was supported in part by a postdoctoral fellowship from the Group Health Foundation. 

The study examined the variation in devices used (desktop/laptop computer, mobile device, or both device types) by patients of different racial/ethnic backgrounds to access the online patient portal.

Racial/Ethnic Variation in Devices Used to Access Patient Portals

 To determine whether comorbidity information derived from electronic health record (EHR) problem lists is accurate.

 Retrospective cohort study of 1596 men diagnosed with prostate cancer between 1998 and 2004 at 2 Southern California Veterans Affairs Medical Centers with long-term follow-up. 

 We compared EHR problem list—based comorbidity assessment with manual review of EHR free-text notes in terms of sensitivity and specificity for identification of major comorbidities and Charlson Comorbidity Index (CCI) scores. We then compared EHR-based CCI scores with free-text–based CCI scores in prediction of long-term mortality. 

 EHR problem list—based comorbidity assessment had poor sensitivity for detecting major comorbidities: myocardial infarction (8%), cerebrovascular disease (32%), diabetes (46%), chronic obstructive pulmonary disease (42%), peripheral vascular disease (31%), liver disease (1%), and congestive heart failure (23%). Specificity was above 94% for all comorbidities. Free-text–based CCI scores were predictive of long-term other-cause mortality, whereas EHR problem list–based scores were not. 

 Inaccuracies in EHR problem list—based comorbidity data can lead to incorrect determinations of case mix. Such data should be validated prior to application to risk adjustment. 

Electronic health record (EHR)-based data are increasingly being used for purposes of risk adjustment, but our findings show that the Veterans Affairs EHR problem list led to incorrect determinations of case mix and did not predict survival, in contrast to chart-based comorbidity assessment. Given the increasing importance of EHR data, we believe that: 

EHR-based comorbidity data should be validated prior to application to risk adjustment. 

Other sources of comorbidity information, such as patient-reported measures of health status or natural language processing—derived data, may be considered for risk adjustment in comparisons of performance.

Recent study results highlight the inconsistency of different sources of data (eg, registries, claims, the electronic health record [EHR]) for identifying basic health information, such as major comorbidities.

 For individual physicians, these inconsistencies are less relevant because they have the opportunity to confirm this information directly with the patient. However, when used for risk adjustment for purposes of performance assessment,

 incorrect data may lead to misclassification and unfair comparisons. This is a major concern for health systems participating in alternative payment models, which base some portion of reimbursement on risk-adjusted quality outcomes.

 Comorbidity is a key component of the risk adjustment needed for fair comparisons of measures of quality, as comorbid disease burden affects readmissions,

 Accurately quantifying comorbidity requires varying degrees of detail regarding number, severity, or types of conditions, depending on the measure used.

 All of these measures require robust data sources to identify the presence or absence of each of the included comorbid conditions. 

Whereas comorbidity data from inpatient medical records are reviewed by trained coders, outpatient records may be less reliable,

 as they often rely on “problem lists” in the EHR to identify the index condition. The problem list is a compilation of patient diagnoses entered by clinicians during patient encounters and updated at varying intervals. With increasing numbers of institutions and office-based practices using EHRs to store patient data, interest has grown in utilizing these lists as a source of comorbidity data.

 However, it is unclear whether the data in these lists are sufficiently accurate to assess patients’ total comorbid disease burden. Recent studies have attempted to validate the accuracy of the problem list by comparing it with other diagnosis lists or with short-term future proximate outcomes, such as glycated hemoglobin.

 However, the most appropriate metric for assessment of validity of these lists is long-term mortality, especially in an elderly population; a longer list of major comorbidities should be strongly associated with higher mortality. 

In this study, we compared the ability of Charlson Comorbidity Index (CCI) scores derived from the Veterans Affairs (VA) problem list to predict mortality in an elderly population with a gold standard for comorbidity assessment, manual abstraction directly from the physician’s free-text notes. We captured mortality over a 10-year period, which was long enough to reveal the impact of even minor comorbidities over time. Because the problem list is not actively maintained, we hypothesized that the problem list would provide poor accuracy in identifying comorbidities and would poorly predict survival compared with free-text—based assessment. 

We used the California Cancer Registry to identify men newly diagnosed with prostate cancer at the Greater Los Angeles and Long Beach VA Medical Centers between 1998 and 2004 (N = 1915). We reviewed EHRs for sociodemographic, tumor risk, comorbidity, and survival data and identified all men with sufficient data to determine comorbidity and survival (n = 1596). Institutional review board approval was granted by the University of California, Los Angeles, and both VA Medical Centers. 

 We assessed comorbid disease burden at the time of prostate cancer diagnosis using 2 sources of data: 1) the EHR free-text notes record and 2) the EHR problem list. The interdisciplinary EHR free-text notes record contained outpatient and inpatient notes from all clinical encounters. Data from the medical record within 12 months of the diagnosis of prostate cancer were used for free-text—based comorbidity assessment. Comorbidities were coded according to the definitions originally indicated by Charlson et al,

 and age-unadjusted CCI scores were calculated. Because free-text—based comorbidity assessment was conducted primarily by 1 author (TJD), reliability was assessed on a random 5% subset of the sample by a separate author (GA). Interrater agreement in Charlson scores was 77.5% and the associated kappa statistic was 0.67. 

The VA problem list was populated and updated by clinicians variably as they deemed appropriate; diagnoses were coded by 

International Classification of Diseases, Ninth Revision

) codes and dates and entered into a retrievable database. Comorbidities added to the problem list up to 12 months after the diagnosis of prostate cancer were used for EHR problem list—based comorbidity assessment. Comorbidities were coded according to the claims-based definitions indicated by Deyo et al,

 and age-unadjusted Deyo-CCI scores were calculated. Neither prostate cancer nor any complications of prostate cancer were included in comorbidity scoring for either comorbidity assessment method.

 Age at diagnosis was coded as a continuous variable. Race/ethnicity was coded as Caucasian, African American, Hispanic, or other. Tumor characteristics included prostate-specific antigen (PSA), Gleason sum, and clinical tumor (T), node, and metastasis stage at diagnosis. Categories for PSA, Gleason sum, and clinical T stage were defined by the widely accepted D’Amico criteria, which have been shown to predict overall and cancer-specific mortality.

Survival was measured from date of treatment until date of death. We determined date of death using a combination of the medical record and the Social Security Death Index. Cause of death was determined using the medical record with an algorithm that has been previously described.

We determined the prevalence of 7 major comorbidities using free-text—based and EHR problem list–based comorbidity assessment. Sensitivity and specificity for identification of major comorbidities were calculated, using free-text–based assessment as the gold standard. Agreement between assessment methods was ascertained for each comorbidity using Cohen’s kappa statistic. 

We compared mean CCI scores based on the problem list and on free-text notes using a paired 

 test for difference in means. Continuous and categorical (0, 1, 2, and ≥3) versions of CCI scores for each assessment method were also compared using Pearson’s correlation and Cohen’s kappa, respectively. 

Multivariable competing risks regression as described by Fine and Gray

 was used to compare prediction of survival by CCI scores derived from each comorbidity assessment method. We calculated subhazard and cumulative incidence of non—prostate cancer mortality by CCI scores for each comorbidity assessment method, treating prostate cancer as a competing risk. All multivariable regression models were adjusted for age, race, VA site, clinical stage, Gleason score, PSA, and treatment type. 

We then conducted a sensitivity analysis to determine if year of diagnosis affected our results, as use of the EHR problem list may have changed over time. We subdivided our group into those diagnosed in 2001 or earlier (n = 810) and 2002 or later (n = 786) and repeated our analyses. 

<.05 to denote statistical significance, and all tests were 2-sided. All statistical analyses were performed in Stata, version 12.0 (Stata, Inc; College Station, Texas). 

The majority of the sample (N = 1596) comprised white (44%) and African American men (37%). Approximately one-half of the sample was 65 years or younger, and most had early-stage prostate cancer (

EHR problem list—based comorbidity assessment had poor sensitivity but high specificity for identification of common major comorbidities (

). Sensitivity values for EHR problem list—based assessment (using free-text–based assessment as the gold standard) ranged from 8% for myocardial infarction to 46% for diabetes. Specificity was above 94% for all comorbidities. Agreement between EHR problem list–based and free-text–based comorbidity assessment was poor for all major comorbidities, with kappa ranging from 0.02 to 0.44. Results did not change after subdividing our group by those diagnosed before or after 2001.

Comparison of the CCI scores based on the EHR problem list and on free text showed that EHR problem list—based assessment underestimated comorbidity burden (

). Agreement across all scores was 53% (840/1596). Pearson correlation for continuous scores was 0.3, and kappa for categorical scores was 0.2. Among scores that were discordant, 82% (627/765) were higher using free-text—based compared with EHR problem list–based comorbidity assessment. Mean free-text–based and EHR problem list–based CCI scores were 1.1 (95% CI, 1.04-1.19) and 0.5 (95% CI, 0.46-0.56), respectively. Free-text–based scores were significantly higher by a mean of 0.6 points (95% CI, 0.53-0.67; 

Competing risks regression analysis showed that free-text—based CCI scores predicted noncancer mortality, whereas EHR problem list–based scores did not. Higher free-text–based CCI scores were associated with a graduated increase in hazard of other-cause mortality (

), and 10-year cumulative incidence of noncancer mortality associated with chart-based scores was 41%, 57%, 64%, and 83% for CCI scores of 0, 1, 2, and ≥3, respectively (

). Higher EHR problem list—based CCI scores were not associated with increased mortality risk (Table 4), and EHR problem list–based CCI scores did not discriminate 10-year cumulative incidence of noncancer mortality: 55%, 64%, 62%, and 58% for CCI scores of 0, 1, 2, and ≥3, respectively (

). Results did not change after subdividing our group by those diagnosed before or after 2001.

Despite interest in capitalizing on readily available problem list data in the EHR for purposes of risk adjustment, our findings suggest that these data should be validated prior to application to performance assessment. The sensitivity of the VA problem list for identifying common major comorbidities was poor, ranging from 1% to 46%, compared with manual free-text note abstraction. This lack of accuracy led to incorrect risk adjustment and would lead to unfair comparisons of clinicians’ quality of care if applied directly to risk adjustment for performance assessment. These data may be generalizable to more contemporary EHRs because comorbidity information in problem lists and outpatient claims still is not actively maintained. Absent a valid method of adjustment for comorbidity, it is not possible to confidently distinguish between physicians or groups who provide poor care and those who disproportionately see patients with greater disease burden. Because measures of quality of care are now being tied to compensation in programs like value-based purchasing,

 the stakes are higher and the consequences of errors in performance assessment are much more substantial. These errors are magnified when the units of comparison are smaller (eg, physician groups or individual physicians).

Previous studies have shown that failure to adjust for case mix can alter quality rankings.

 For example, in a comparison of functional outcomes at 4 hospitals 1 year after total hip replacement, significant differences in institutional outcomes were apparent before adjustment for comorbid disease status but not after.

 In our study, CCI scores from free-text—based assessment predicted mortality, whereas EHR problem list–based scores did not, likely due to the latter’s poor sensitivity for detecting major comorbid diseases. For clinical purposes, such as risk adjustment of quality scores, versus billing and utilization purposes, comorbidity scores relying on EHR diagnoses could be misleading. 

Inaccuracy in identification of major comorbid conditions and its impact on risk adjustment could have significant financial consequences for physicians and physician groups participating in accountable care organizations (ACOs). ACOs, such as those participating in the Medicare Shared Savings Program, require participating organizations to report certain quality measures to CMS.

 Only providers who meet the program’s quality performance benchmarks will be eligible to receive a share of cost savings, which can be substantial. For example, in the Physician Group Practice Demonstration, 7 of 10 groups received $108 million in shared savings payments over the course of the demonstration.

 Providers who do not meet quality benchmarks will be subject to penalties. Although early ACOs have started with fairly undemanding process benchmarks for quality, such as the Healthcare Effectiveness Data and Information Set measures for chronic disease management, which are met by a national average of 79% to 80% of groups,

 quality benchmarks will likely become more stringent over time, as evidenced by the inclusion of all-cause readmission in quality scoring in the third year of participation in the Medicare Shared Savings Program.

 There is an almost 2-fold variation in readmissions across different areas of the United States,

Electronic health record (EHR)-based comorbidity assessment had low sensitivity for identifying major comorbidities and poorly predicted survival. EHR-based comorbidity data require validation prior to application to risk adjustment.

January 2018

Hospitalized Patients' and Family Members' Preferences for Real-Time, Transparent Access to Their Hospital Records

Racial/Ethnic Variation in Devices Used to Access Patient Portals

Electronic Health Record Problem Lists: Accurate Enough for Risk Adjustment?

Electronic Sharing of Diagnostic Information and Patient Outcomes

Measuring Overuse With Electronic Health Records Data

The Health Information Technology Special Issue: Has IT Become a Mandatory Part of Health and Healthcare?

A Cost-Effectiveness Analysis of Cardiology eConsults for Medicaid Patients

Electronic Health Record "Super-Users" and "Under-Users" in Ambulatory Care Practices

Hospital Participation in Meaningful Use and Racial Disparities in Readmissions

Bridging the Digital Divide: Mobile Access to Personal Health Records Among Patients With Diabetes