Grant Martsolf, PhD, MPH, RN

 To evaluate structural capabilities in primary care practices employing nurse practitioners (NPs) and test whether they differ across health professional shortage areas (HPSAs) and non-HPSAs.

 Secondary analysis of cross-sectional survey data and health care workforce data from 2018-2019.

 We computed bivariate analyses and multivariable adjusted regression models to evaluate differences in NP characteristics and practice characteristics and to determine the odds of having particular structural capabilities in HPSA practices compared with non-HPSA practice.

 The majority of NPs worked in HPSA practices (61%). We found statistically significant differences in NP educational degrees, practice certifications, and structural capabilities between HPSAs and non-HPSAs. Care coordination was 77% more likely to be delivered in HPSA practices compared with non-HPSA practices (odds ratio, 1.77; 

 Expanding care coordination may be beneficial for HPSA populations with high rates of morbidity and socioeconomic needs. Future research is needed to understand how the NP workforce may be optimized to meet the growing primary care demands in underserved areas. 

Nurse practitioners (NPs) are more likely to deliver care coordination in health professional shortage areas (HPSAs) compared with non-HPSAs and are meeting the growing demand for primary care in underserved areas.

NPs deliver specific structural capabilities (ie, practice features or infrastructure) related to chronic care to support the complex and chronically ill populations in HPSAs.

Without expansion of scope-of-practice laws, workforce shortages will remain a perennial challenge.

HPSA criteria do not include the availability of additional providers such as NPs, which may complicate the accuracy of identifying and evaluating care in shortage areas.

Providing high-quality primary care has proved to be challenging as the United States faces a national shortage of primary care physicians compounded by growing rates of an aging and chronically ill population.

 Currently, 80 million Americans reside in geographic areas known as health professional shortage areas (HPSAs), which are disproportionately affected by a lack of supply of primary care physicians relative to the general population.

 Improving primary care access in HPSAs is critical because maldistribution of the health care workforce is linked to poor health outcomes including disease severity, quality of life, and life expectancy.

 Indeed, patient populations residing in HPSAs have higher prevalence of diabetes, hypertension, obesity, and smoking compared with patients not living in these areas. Provider shortages in HPSAs can also exacerbate existing socioeconomic barriers to obtaining care, such as education, housing, or transportation needs.

Deploying the nurse practitioner (NP) workforce has been identified as an optimal strategy by policy makers and administrators to meet the growing demands in primary care. NPs are the fastest-growing primary care workforce in the United States,

 and approximately 89% of NPs are equipped to deliver primary care.

 NPs demonstrate equivalent patient outcomes to physicians, including disease-specific physiologic measures, symptom reduction, and acute care utilization.

NPs are specifically equipped to expand primary care in HPSAs for patients with high rates of multimorbidity and complex social needs because NPs are trained in comprehensive and holistic care emphasizing a patient’s broad health needs and their social and emotional well-being.

 Indeed, NPs are significantly more likely to care for patients with 3 or more chronic conditions compared with physicians (25.9% vs 20.8%)

 and to deliver chronic disease services such as disease education and counseling.

 Further, in US states with full scope-of-practice regulation—meaning NPs can independently evaluate, diagnose, interpret tests, and treat patients—NPs are more likely to practice and reside in HPSAs compared with physicians.

 Full scope-of-practice regulation for NPs is also associated with an approximate 30% increase in yearly checkups in HPSAs.

However, little is known about the practice infrastructure or integrated features (ie, structural capabilities) used by NPs to enhance primary care delivery in HPSAs. Previous research demonstrates that practice structural capabilities can improve primary care access through extended practice hours and delivery of chronic care through reminders for provider workflows, or care coordination.

 Care coordination, for example, consists of the integration of personnel or activities used to manage patient care across the health care spectrum and has been shown to be associated with lower medical expenditures, inpatient hospitalizations, emergency department (ED) visits, and 30-day readmission rates.

 Other structural capabilities such as availability of chronic disease registries—designed to support providers in managing patients with chronic illness through tracking systems, clinician reminders, and checklists

—have been shown to improve patient outcomes and to support practices in achieving the standard of care for ongoing chronic care.

Despite such findings, it is unclear whether HPSA practices that are employing the growing NP workforce implement specific structural capabilities to meet the needs of their complex and chronically ill patient populations. Such evidence is needed to understand how the NP workforce may be optimized to expand access to primary care in underserved areas. To fill this gap, our study assessed primary care practice structural capabilities in practices employing NPs and tested whether they differ across HPSAs and non-HPSAs.

This study was a secondary analysis of cross-sectional data from 2 sources: (1) survey data collected from primary care NPs in 2018-2019 (principal investigator: Poghosyan; R01MD011514) on NP characteristics, practice setting, and structural capabilities; and (2) publicly available data from the Health Resources and Services Administration (HRSA) from 2018-2019 on HPSA designation.

Survey data were collected from primary care NPs as part of a large study focused on racial and ethnic disparities in chronic disease outcomes and NP practice. Researchers sent surveys to 10,237 NPs in primary care settings across 6 geographically diverse states. NPs were identified through the OneKey database from IQVIA, which includes the most complete information on office-based providers in the United States.

 Using a modified Dillman method, 3 rounds of mail and online surveys were sent out to eligible NPs with reminder postcards and phone calls to nonresponders.

 Overall, 1244 NPs in 1109 practices completed and returned the surveys, with a response rate of 31.2%. A nonresponse analysis was conducted and determined to have low bias.

Data on HPSA designation were collected by researchers for the large study. HPSA designation was obtained from the Primary Care Service Area files, which contain data on primary care availability in communities and are maintained by the Dartmouth Institute. Data on HPSA designation are updated regularly and made publicly available by HRSA.

Survey data on NP practices were merged with data on HPSA designation using zip code information available both in the NP survey and in the Primary Care Service Area files. The NP survey provided either 5-digit or 4-digit zip codes on NP practice location, and the Primary Care Service Area files provided 9-digit zip codes to identify HPSA geographic areas. Crosswalk files from the US Department of Housing and Urban Development were used to link differing zip code levels using an incident fraction for each 9-digit and 5-digit zip code to account for differences in geographic scales.

We extracted data from the parent study from Arizona and Washington because these states have full scope-of-practice regulation. NPs are more likely to practice and reside in HPSAs in states with full scope-of-practice regulation.

 We sampled surveys from 366 NPs across 269 unique practices in Arizona (46%) and Washington (54%).

 Our independent variable of interest was HPSA designation calculated on a range from 0 to 25. Criteria for designation are (1) the population to provider ratio (10-point maximum), (2) percentage of the population below 100% of the federal poverty level (5-point maximum), (3) travel time to the nearest source of care outside the HPSA (5-point maximum), and (4) infant health index (5-point maximum).

 Practices without any HPSA score (HPSA score 0 or not applicable) were coded as “non-HPSA,” and practices with any score (HPSA score 1-25) were coded as “HPSA,” indicating geographic areas or populations with inadequate access to primary care.

 Structural capabilities were obtained from the Structural Capability Index (SCI) contained within NP survey data. The SCI is a validated measure used to evaluate primary care practice attributes linked with high-quality care delivery.

 Previous research has used the SCI to explore structural capabilities in medical homes and the impact of various structural capabilities on patient outcomes, patient satisfaction, and quality of care.

We selected 4 subscales on the SCI that have been shown to expand access to care and improve ongoing disease management: (1) 

 (3 items), shown to improve patient satisfaction and enhance timely communication between patients and providers

 (7 items), associated with lower medical expenditures, hospitalizations, ED visits, and readmission rates

 (1 item), shown to improve patient outcomes and support practices to achieve the standard for chronic care in diabetes and hypertension

 (2 items), which is associated with reduced ED utilization, lower expenditures, and lower rates of unmet medical needs.

The majority of structural capabilities were measured on subscales with binary (yes/no) responses in which NPs reported on the presence or absence of each structural capability. We dichotomized subscales with multiple items by operationalizing the entire subscale as present if more than 50% of items were reported as present. This method of standardization is consistent with prior research.

 provides the corresponding survey items for each capability evaluated.

To isolate the relationship between HPSA and practice capabilities, we controlled for NP demographics and characteristics including age, gender, race, education (ie, highest educational degree received), time employed (ie, time working at current practice), and certification. Practice characteristics included practice type (eg, physician practice, community health center, hospital-based clinic) and practice size (eg, number of NPs, physicians, or physician assistants).

Descriptive statistics were computed to describe the characteristics of NP respondents and their practices. Bivariate χ

 analyses were computed to evaluate differences in NP and practice characteristics across HPSAs and non-HPSAs. We built multivariate logistic regression models to evaluate the odds of having a structural capability in a HPSA practice compared with a non-HPSA practice. Multicollinearity was assessed between covariates by calculating the variance inflation factor and were determined to be acceptable. Final models adjusted for NP demographics and practice characteristics. We also adjusted for the clustering of NPs within practices to ensure that the sample variance was not underestimated and to decrease the potential for type 1 error. Analyses were performed using RStudio version 1.3.

 presents the characteristics of NPs and primary care practices in our sample in HPSAs and non-HPSAs. Overall, 366 NPs responded in 269 distinct practices. The majority of NPs (61%) practiced in areas designated as HPSAs. Demographic characteristics of NPs in HPSA practices were generally similar to those in non-HPSAs. The mean age of NPs in our sample was 50 years. In both HPSA and non-HPSA practices, the plurality of NPs were female (85%-87%), aged between 31 and 44 years, and had worked at their current primary care practice for 5 years or fewer. HPSAs had similar racial composition to non-HPSAs; in both HPSAs and non-HPSAs, the largest percentage of NPs was White or Caucasian (87% in HPSAs vs 84% in non-HPSAs) and the second largest was Asian (6.8% in HPSAs vs 10% in non-HPSAs).

There were statistically significant differences in educational degrees and practice certifications between NPs in HPSAs and those in non-HPSAs. NPs in HPSAs were significantly more likely to have a greater distribution of specialties such as hospice, midwifery, wound care, or HIV medicine compared with NPs in non-HPSAs (6.7% vs 2.8%; 

 < .05). In HPSAs, NPs were less likely to have only an adult certification (11% vs 18%; 

 < .05). In HPSA practices, a lower percentage of NPs had a doctorate degree (18% vs 22%; 

 < .05). All NPs in non-HPSAs had graduate degrees at either the master’s or doctoral level. Practice size was similar in HPSAs and non-HPSAs. The plurality of practices in both HPSAs and non-HPSAs (37%) were run by more than 7 primary providers including NPs, physicians, and physician assistants.

The prevalence of structural capabilities across all primary care practices in our sample is demonstrated in 

 and calculated based on the percentage of NP respondents. The most prevalent structural capability across practices was chronic disease registries (65%). The least prevalent practice capability was access to after-hours care during the week (24%) and on the weekends (30%). Only 43% of practices offered care coordination.

Results from our regression models show significant differences in structural capabilities across HPSAs and non-HPSAs (

). NPs who practiced in HPSAs were 68% more likely to have care coordination (odds ratio [OR], 1.68; 

 < .05), even after controlling for differences in individual NP and practice characteristics (OR, 1.77; 

 < .05). Although the difference was not significant, NPs who practiced in HPSAs were also more likely to report implementing chronic disease registries. Both unadjusted and adjusted regressions demonstrated that NPs in HPSA practices were 26% more likely to implement chronic disease registries (OR, 1.26; 

There was no significant difference in shared communication systems or after-hours care across HPSA and non-HPSA practices, but NPs who practiced in HPSAs were slightly less likely to have shared communication compared with NPs in non-HPSAs (unadjusted OR, 0.99; 

 = .95). On the other hand, NPs in HPSA practices were slightly more likely to offer after-hours care even after accounting for differences in practice size and NP demographics (OR, 1.07; 

 = .81). However, neither of these relationships were statistically significant.

Using cross-sectional survey data from 2 states, this study explored the association between primary care practice structural capabilities and HPSA designation. We found that NPs were significantly more likely to deliver care coordination in practices located in HPSAs compared with non-HPSAs. Delivering care coordination has been shown to improve disease management and reduce ED utilization specifically for patients with complex needs and multimorbidity,

 and care coordination may be used more frequently in HPSA practices to support their complex and chronically ill populations. Further, care coordination is also more likely to be primarily delivered by NPs, underscoring the key role of NPs in enhancing primary care delivery.

Although the difference was not statistically significant, our study also demonstrated that NPs in HPSA practices are more likely to implement disease registries. Disease registries and reminders for chronic care are associated with improved patient outcomes including reaching target adherence measures for diabetes and achieving clinical practice guidelines for care,

 as well as lower per-beneficiary spending.

 Additionally, disease registries may be used on paper rather than implementing new software or electronic health records.

 Thus, disease registries show promise in improving chronic care at low cost to practices.

We evaluated states with full scope-of-practice laws to assess how to optimize NPs in underserved areas when they are able to practice as independent providers. Consistent with research that found that NPs are more likely to practice in HPSAs in states with full scope-of-practice regulations,

 we found that the majority of NPs in our sample (61%) were working in HPSA practices. However, the formula used to designate primary care HPSAs does not take into account the availability of additional primary care providers such as NPs or physicians, which may complicate the accuracy of evaluating differences across HPSA designations.

Consequentially, in this study the extent of differences across HPSA designation in NP characteristics and structural capabilities was largely insignificant. Existing research similarly demonstrates small and insignificant differences in patient health status and access to care across HPSAs and non-HPSAs.

 We recommend that future research continues to evaluate patient and provider outcomes across HPSA designation to test the sensitivity of HPSA criteria and designation status. Further, refining the HPSA criteria to include availability of the NP workforce would be an important contribution to inform policy and demonstrate more accurately how NPs are meeting the primary care needs of underserved areas.

Substantial evidence exists demonstrating the positive impact of care coordination on a variety of indicators including improved quality of life, decreased medical expenditures, and lower rates of inpatient hospitalizations.

 Yet only 43% of NPs in our sample reported delivering care coordination at their practices. Two strategies may be used to support care coordination implementation: First, enhancing practice infrastructure through dedicated personnel, electronic medical records, or psychosocial resources can be useful to facilitate effective care coordination delivery.

 Second, emphasizing the use of chronic care management codes from CMS can incentivize practices by reimbursing for care management or coordination for Medicare beneficiaries.

This study was conducted in states with full scope-of-practice regulations for NPs, and many of the structural capabilities that we studied (ie, care coordination and disease management) are primarily delivered by NPs.

 We chose full scope-of-practice states to control for regulatory differences. Consequentially, we were unable to assess whether variation in scope-of-practice regulation (ie, full vs reduced vs restricted practice laws) modifies the relationship between HPSA designation and the presence of structural capabilities. Thus, it is unclear whether the extent of regulation impacts the presence or delivery of structural capabilities. Future research should consider how restrictive scope-of-practice regulation may affect implementation of structural capabilities in order to make actionable policy recommendations for optimizing NP care delivery and improving primary care access in underserved areas.

We sampled NPs from 2 states, limiting the extent to which these findings can be generalized to other states. Self-reported survey data are subject to self-report bias. To control biases, validated tools were used with rigorous methodology throughout survey development and data collection. We were unable to assess whether there were differences in the quality of structural capabilities that may exist among practices. For example, NPs reported only on the presence of structural capabilities within their practices, rather than their actual or perceived efficacy. We also did not include whether NPs were practicing in teams or independently and how the presence of teams may affect the implementation of structural capabilities within a practice. Lastly, the use of cross-sectional data limits our ability to infer causation.

We found statistically significant differences in NP educational degrees, practice certifications, and structural capabilities between HPSA and non-HPSAs. NPs in HPSA-designated primary care practices were significantly more likely to have care coordination compared with those in practices that are located in areas with adequate access to primary care. Expanding care coordination may be beneficial for complex and chronically ill patients residing in HPSAs. Future research is needed to understand how to optimize the NP workforce and implementation of structural capabilities to meet the growing demands for primary care in underserved areas. 


 School of Nursing (AB, AES, VH, JL, LP) and Mailman School of Public Health (LP), Columbia University, New York, NY; University of Pittsburgh (GM), Pittsburgh, PA.

 The data were produced from a project supported by the Agency for Healthcare Research and Quality (AHRQ), PI: Poghosyan, R01MD011514.

 Dr Schlak is a postdoctoral research fellow funded by the National Institute of Nursing Research Training Grant (T32-NR014205, Stone) during the conduct of the study. Dr Poghosyan received AHRQ grant R01MD011514. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (AB, GM, AES, VH, LP); acquisition of data (GM, JL, LP); analysis and interpretation of data (AB, AES, JL); drafting of the manuscript (AB, AES, VH); critical revision of the manuscript for important intellectual content (AB, AES, VH, LP); statistical analysis (AB, JL); obtaining funding (GM); and supervision (GM, JL, LP).

 Ani Bilazarian, PhD, BSN, RN, Columbia University School of Nursing, 560 W 168th St, New York, NY 10032. Email: ab4797@cumc.columbia.edu.

1. Duchovny N, Trachtman S, Werble E. Projecting demand for the services of primary care doctors. Congressional Budget Office. May 2017. Accessed February 5, 2021. https://www.cbo.gov/system/files/115th-congress-2017-2018/workingpaper/52748-workingpaper.pdf

2. Raghupathi W, Raghupathi V. An empirical study of chronic diseases in the United States: a visual analytics approach to public health. 

. 2018;15(3):431. doi:10.3390/ijerph15030431

3.HPSA Find. Health Resources and Services Administration. Accessed January 5, 2021. https://data.hrsa.gov/tools/shortage-area/hpsa-find

4. Allen NB, Diez-Roux A, Liu K, Bertoni AG, Szklo M, Daviglus M. Association of health professional shortage areas and cardiovascular risk factor prevalence, awareness, and control in the Multi-Ethnic Study of Atherosclerosis (MESA). 

. 2011;4(5):565-572. doi:10.1161/CIRCOUTCOMES.111.960922

5. Basu S, Berkowitz SA, Phillips RL, Bitton A, Landon BE, Phillips RS. Association of primary care physician supply with population mortality in the United States, 2005-2015. 

. 2019;179(4):506-514. doi:10.1001/jamainternmed.2018.7624

6. Streeter RA, Snyder JE, Kepley H, Stahl AL, Li T, Washko MM. The geographic alignment of primary care health professional shortage areas with markers for social determinants of health. 

. 2020;15(4):e0231443. doi:10.1371/journal.pone.0231443

7. Auerbach DI, Staiger DO, Buerhaus PI. Growing ranks of advanced practice clinicians — implications for the physician workforce. 

. 2018;378(25):2358-2360. doi:10.1056/nejmp1801869

8.Nurse practitioners in primary care. American Association of Nurse Practitioners. Accessed January 6, 2021. https://www.aanp.org/advocacy/advocacy-resource/position-statements/nurse-practitioners-in-primary-care

9. Buerhaus P, Perloff J, Clarke S, O’Reilly-Jacob M, Zolotusky G, DesRoches CM. Quality of primary care provided to Medicare beneficiaries by nurse practitioners and physicians. 

. 2018;56(6):484-490. doi:10.1097/MLR.0000000000000908

10. Kuo YF, Chen NW, Baillargeon J, Raji MA, Goodwin JS. Potentially preventable hospitalizations in Medicare patients with diabetes: a comparison of primary care provided by nurse practitioners versus physicians. 

. 2015;53(9):776-783. doi:10.1097/MLR.0000000000000406

11. Kurtzman ET, Barnow BS. A comparison of nurse practitioners, physician assistants, and primary care physicians’ patterns of practice and quality of care in health centers. 

. 2017;55(6):615-622. doi:10.1097/MLR.0000000000000689

12. Yang Y, Long Q, Jackson SL, et al. Nurse practitioners, physician assistants, and physicians are comparable in managing the first five years of diabetes. 

. 2018;131(3):276-283.e2. doi:10.1016/j.amjmed.2017.08.026

13. Grant J, Lines L, Darbyshire P, Parry Y. How do nurse practitioners work in primary health care settings? a scoping review. 

. 2017;75:51-57. doi:10.1016/j.ijnurstu.2017.06.011

14. Fraze TK, Briggs ADM, Whitcomb EK, Peck KA, Meara E. Role of nurse practitioners in caring for patients with complex health needs. 

. 2020;58(10):853-860. doi:10.1097/MLR.0000000000001364

15. Lin SX, Gebbie KM, Fullilove RE, Arons RR. Do nurse practitioners make a difference in provision of health counseling in hospital outpatient departments? 

. 2004;16(10):462-466. doi:10.1111/j.1745-7599.2004.tb00425.x

16. Ritsema TS, Bingenheimer JB, Scholting P, Cawley JF. Differences in the delivery of health education to patients with chronic disease by provider type, 2005-2009. 

17. DePriest K, D’Aoust R, Samuel L, Commodore-Mensah Y, Hanson G, Slade EP. Nurse practitioners’ workforce outcomes under implementation of full practice authority. 

. 2020;68(4):459-467. doi:10.1016/j.outlook.2020.05.008

18. Xue Y, Kannan V, Greener E, et al. Full scope-of-practice regulation is associated with higher supply of nurse practitioners in rural and primary care health professional shortage counties. 

. 2018;8(4):5-13. doi:10.1016/S2155-8256(17)30176-X

19. Traczynski J, Udalova V. Nurse practitioner independence, health care utilization, and health outcomes. 

. 2018;58:90-109. doi:10.1016/j.jhealeco.2018.01.001

20. Friedberg MW, Coltin KL, Safran DG, Dresser M, Zaslavsky AM, Schneider EC. Associations between structural capabilities of primary care practices and performance on selected quality measures. 

. 2009;151(7):456-463. doi:10.7326/0003-4819-151-7-200910060-00006

21. Martsolf GR, Ashwood S, Friedberg MW, Rodriguez HP. Linking structural capabilities and workplace climate in community health centers. 

. 2018;55:46958018794542. doi:10.1177/0046958018794542

22. Berkowitz SA, Parashuram S, Rowan K, et al; Johns Hopkins Community Health Partnership (J-CHiP) Team. Association of a care coordination model with health care costs and utilization: the Johns Hopkins Community Health Partnership (J-CHiP). 

. 2018;1(7):e184273. doi:10.1001/jamanetworkopen.2018.4273

23. Powers BW, Modarai F, Palakodeti S, et al. Impact of complex care management on spending and utilization for high-need, high-cost Medicaid patients. 

. 2020;26(2):e57-e63. doi:10.37765/ajmc.2020.42402

24. Burton RA, Zuckerman S, Haber SG, Keyes V. Patient-centered medical home activities associated with low Medicare spending and utilization. 

. 2020;18(6):503-510. doi:10.1370/afm.2589

25. Peterson KA, Carlin C, Solberg LI, Jacobsen R, Kriel T, Eder M. Redesigning primary care to improve diabetes outcomes (the UNITED Study). 

. 2020;43(3):549-555. doi:10.2337/dc19-1140

26. DesRoches CM, Barrett KA, Harvey BE, et al. The results are only as good as the sample: assessing three national physician sampling frames. 

. 2015;30(suppl 3):S595-S601. doi:10.1007/s11606-015-3380-9

27. Hoddinott SN, Bass MJ. The Dillman total design survey method. 

28. Harrison JM, Germack HD, Poghosyan L, D’Aunno T, Martsolf G. Methodology for a six-state survey of primary care nurse practitioners. 

. 2021;69(4):609-616. doi:10.1016/j.outlook.2021.01.010

29. Din A, Wilson R. Crosswalking zip codes to census geographies: geoprocessing the U.S. Department of Housing & Urban Development’s zip code crosswalk files. 

. 2020;22(1):293-314. https://www.jstor.org/stable/26915499

30. Friedberg MW, Safran DG, Coltin KL, Dresser M, Schneider EC. Readiness for the patient-centered medical home: structural capabilities of Massachusetts primary care practices. 

. 2009;24(2):162-169. doi:10.1007/s11606-008-0856-x

31. Martsolf GR, Kandrack R, Baird M, Friedberg MW. Estimating associations between medical home adoption, utilization, and quality: a comparison of evaluation approaches. 

. 2018;56(1):25-30. doi:10.1097/MLR.0000000000000842

32. Liederman EM, Morefield CS. Web messaging: a new tool for patient-physician communication. 

. 2003;10(3):260-270. doi:10.1197/jamia.M1259

33. Sada YH, Street RL Jr, Singh H, Shada R, Naik AD. Primary care and communication in shared cancer care: a qualitative study. 

34. Hoque DME, Kumari V, Hoque M, Ruseckaite R, Romero L, Evans SM. Impact of clinical registries on quality of patient care and clinical outcomes: a systematic review. 

. 2017;12(9):e0183667. doi:10.1371/journal.pone.0183667

35. Jerant A, Bertakis KD, Fenton JJ, Franks P. Extended office hours and health care expenditures: a national study. 

. 2012;10(5):388-395. doi:10.1370/afm.1382

36. O’Malley AS. After-hours access to primary care practice linked with lower emergency department use and less unmet medical need. 

. 2013;32(1):175-183. doi:10.1377/hlthaff.2012.0494

37. Capp R, Misky GJ, Lindrooth RC, et al. Coordination program reduced acute care use and increased primary care visits among frequent emergency care users. 

. 2017;36(10):1705-1711. doi:10.1377/hlthaff.2017.0612

38. Orzano AJ, Strickland PO, Tallia AF, et al. Improving outcomes for high-risk diabetics using information systems. 

. 2007;20(3):245-251. doi:10.3122/jabfm.2007.03.060185

Articles

Nurse practitioners are increasingly meeting primary care demands in underserved areas and are more likely to deliver structural capabilities related to chronic disease management.

Primary Care Practice Structural Capabilities in Health Professional Shortage Areas

 To understand the views of prominent organizations in the field of healthcare quality on the topic of reporting roll-up measures that combine indicators of multiple, often disparate, dimensions of care to consumers.  

This study used a semi-structured, qualitative interview design.

We conducted 30- to 60-minute semi-structured telephone interviews with representatives of 10 organizations that sponsor public healthcare quality reports and 3 national alliances representing multiple stakeholder groups. We conducted a thematic analysis of interview transcriptions to identify common issues and concerns related to reporting roll-up measures.

Among sponsors reporting roll-up measures, current practices for calculating and reporting these measures are diverse. The main perceived benefit of reporting roll-up measures is that they simplify large amounts of complex information for consumers. The main concern is the potential for consumers to misunderstand the measures and what associated roll-up scores communicate about provider performance. Report sponsors and national alliances feel that more guidance and research on the methods for producing and reporting scores for roll-up measures are needed.

The results of the interviews elucidate the need for research focused on construction and reporting of roll-up measures. Studies are needed to determine if roll-up measures are indeed perceived by consumers as being less complex and easier to understand.  

Some organizations that sponsor public reports of healthcare quality present roll-up measures that combine indicators of multiple, often disparate, dimensions of care. We found that: 

Report sponsors’ practices for calculating and reporting these measures are diverse. 

Report sponsors and national alliances that represent stakeholder groups affected by quality reporting feel that roll-up measures simplify large amounts of complex information for consumers but run the risk of misinterpretation. 

Report sponsors and national alliances indicate a need for more guidance and research on how to produce and report scores for roll-up measures.

A major objective of public reports on healthcare quality is to enable consumers to make well-informed choices about their care. To meet this objective, report sponsors often use strategies intended to reduce the cognitive burden of healthcare quality data and improve consumers’ comprehension and use of the information.

 One strategy is the provision of “roll-up” measures (ie, summary measures of healthcare quality that combine indicators of multiple, often disparate, dimensions of care). For example, a roll-up measure of patient experience with primary care could aggregate composite measures of patient—provider communication, access to care, and interactions with office staff. At a higher level of aggregation, a roll-up measure of overall hospital quality could combine measures of patient experience, clinical processes, and patient outcomes. The most prominent use of roll-up measures within healthcare is CMS’ Five-Star Quality Rating System, which assigns an overall rating of 1 to 5 stars to providers and plans based on multiple quality dimensions.

 Roll-up measures could also aggregate component measures at other levels. 

Publicly reporting roll-up measures offers advantages to consumers insofar as the roll-ups are created scientifically and grounded in high-quality research. These advantages include offering consumers a summary of multiple dimensions of care and, in many cases, a measure of healthcare quality that has superior reliability.

 However, this strategy is not without controversy:  roll-up measures may obscure important nuances of quality and not align with consumer preferences or information needs. Also, the calculation of roll-up scores relies on assumptions about the relative importance of each component measure.

We aimed to understand how prominent organizations in the field of healthcare quality view the reporting of roll-up measures and obtain examples of current approaches to reporting roll-up measures. To that end, this study gathered information from representatives of report sponsors and national healthcare alliances on current practices and positive and negative perceptions with regard to creating and reporting roll-up measures to consumers. We sought to identify these organizations’ concerns given the current interest in reporting these measures to consumers and to determine the need for additional evidence to address these issues.

In this qualitative study, we conducted 30- to 60-minute semi-structured telephone interviews with representatives from 10 organizations that publish comparative reports on healthcare quality (Table 1) and 3 national alliances that bring together multiple stakeholder groups. These alliances—the AQA (formerly the Ambulatory Care Quality Alliance), Consumer-Purchaser Alliance, and the National Quality Forum&mdash;were strategically selected for their expertise in healthcare quality measurement and reporting.

Interviews with report sponsors explored their perceptions of the benefits and drawbacks of reporting roll-up measures, the factors that influenced whether they report these measures, and for those that do report, how they calculate and report these measures to consumers. Interviews with the national alliances focused on whether the reporting of roll-up measures had been discussed within the organization, perceptions of the most important issues or concerns, the types of stakeholders involved in the discussion, and whether they created position statements or recommendations on reporting roll-up measures. Our qualitative approach enabled participants to describe their perceptions and experiences in their own words and was well-suited to eliciting participants’ diverse perspectives.

We used thematic analysis to extract concepts that emerged from multiple interviews.

 Each interview transcription was reviewed by the first author (who also conducted several interviews with report sponsors and all 3 interviews with national alliances). The author extracted key information related to roll-up measure construction and reporting, identified consistent themes across the interviews, and produced short memos describing themes and the evidence for each. After clarifying assumptions and reviewing the themes with the other authors, the author updated the memos to reflect a consensus around the main themes extracted from the data. Key themes were confirmed through an independent review of transcriptions by the fourth author, who did not conduct any of the interviews.

Diversity of Roll-up Measure Reporting Practices

Among sponsors reporting roll-up measures at the time of the interviews, practices for calculating and reporting roll-up measures were diverse. Reporting practices for different providers varied in terms of construction and labeling (Table 2).

 For example, the Wisconsin Department of Employee Trust Funds differentially weighted and combined multiple Consumer Assessment of Healthcare Providers and Systems (CAHPS) and Healthcare Effectiveness Data and Information Set (HEDIS) indicators into a roll-up measure called the “Overall Quality Score” for health plans. In contrast, the Pacific Business Group on Health used an equal-weighting approach to combine several HEDIS measures of medical group performance into the measure, “Medical Group Provides Recommended Care.” 

In the absence of well-established methodological criteria, report sponsors based their decisions about how to calculate and label roll-up scores on information from a variety of sources, including research literature on reporting practices, technical expert guidance, information about other report sponsors’ practices, and the results of psychometric analyses. It is unclear exactly how they incorporated these sources of information into their choices about roll-up measure construction. Some participants reported testing measure labels with consumers or working with advisory groups to construct roll-up measures. All advisory groups included a diverse mix of stakeholders, such as purchasers, providers, state medical associations, and patient advocates. 

The main perceived benefit of reporting roll-up measures is the simplification of large amounts of complex information for consumers. Nearly all participants pointed out that roll-up measures make it easier for consumers to understand and use the information provided in quality reports. For example, one report sponsor representative said, “It’s a huge win for consumers to be able to go in and not have to read through 100 technical measures about hospitals. Even in plain language, 100 measures is a lot of stuff to expect people to go through.” Representatives of the national alliances suggested multiple ways in which reporting roll-up measures could reduce complexity for consumers. One representative asserted that processing roll-up measures requires less effort on the part of the consumer; another argued that reporting these measures allows consumers to gain a comprehensive view of care. In addition to reducing complexity, a third representative suggested that the reporting of roll-up measures could draw attention to the availability of comparative quality information, saying, “…to a large extent, the complexity of existing public reporting has meant that people don’t pay any attention and still ask the neighbors next door, ‘Who do you like?’…so you have to get it to the level that garners attention….” 

However, participants recognized that consumers vary in their need for information and some may want simplified information, but others desire detail (eg, the ability to drill down to scores for the measures that compose the roll-up measures). According to one national alliance representative, high-profile consumer groups argue that when consumers are shown roll-up measures, they should also have the option to view all component measures.

The main concern about reporting roll-up measures is the potential for consumers to misunderstand the measures and what they communicate about provider quality. As one participant noted, if the weighting of individual scores in roll-up measures does not correspond to the relative importance of different aspects of care to consumers, consumers may draw incorrect conclusions about provider performance. Another participant reported that his organization opted against reporting a particular roll-up measure because of uncertainty about how to appropriately weight the component measures. Several report sponsors shared the concern about a potential mismatch between roll-up measures and the salience of individual measures to consumers. One sponsor suggested giving consumers a way to roll up information according to their preferences via an interactive online report.

Several other participants expressed concerns about consumers’ potential misunderstanding of roll-up measures and what they communicate. For example, consumers “might not read the fine print” and, therefore, might miss information about how the roll-up measures were calculated. They may also misunderstand a roll-up measure labeled as “overall quality,” as such a label may imply that all of the important aspects of quality have been measured and are thus represented by the score. In addition, these measures may obscure areas in which providers perform especially well or especially poorly. One report sponsor noted, “That’s always a risk—a person walking away with a generalized sense of performance, but it doesn’t encapsulate the variation of performance of the measures within the composite.”

More guidance and research on the methods for producing and reporting scores for roll-up measures are needed. Several participants expressed a need for more evidence-based guidance on the construction and use of roll-up measures. In the words of one national alliance representative: “I think we could use more science of what logically hangs together, whether there are some things that should never be broken apart, whether there are some things that logically should always hang together.”

We conducted interviews with representatives of report sponsors and national alliances to document attitudes and practices around the calculation and reporting of roll-up measures to healthcare consumers. We identified diverse practices in the construction of roll-up scores; however, our study is limited by its small sample of strategically selected organizations, and we cannot determine the prevalence of roll-up score reporting. The interviews revealed a desire for more guidance and research on the methods for producing and reporting scores for roll-up measures. These findings point to the need for a research agenda focused on developing standards for the construction, calculation, and reporting of roll-up measures that are rooted in empirical data on how constructs assessed by various performance measures relate to each other and desired health outcomes.

One key topic for exploration is the appropriate level of aggregation for roll-up measures. Roll-ups could aggregate measures from a single source (eg, the CAHPS Clinician & Group Survey) or a single domain (eg, clinical quality, patient experience). Alternatively, a single overall roll-up measure, such as the star ratings provided through the CMS Five-Star Rating System, could encompass multiple domains of quality. Although this type of measure may maximize simplicity for consumers, constructing such a score requires strong and nontrivial assumptions about how best to reliably and accurately use a single indicator to communicate the multidimensional construct of healthcare quality.

Although participants perceived that reporting roll-up scores to consumers helps simplify the complex array of healthcare quality information presented in reports, they were concerned about the potential for consumers to misunderstand the aspects of quality captured by roll-up measures and how providers perform on discrete aspects of quality. Whether these beliefs are well-founded is a question worth investigating. Some report sponsors conduct research on roll-up measure construction,

 but the body of published research on consumer use of, and responses to, roll-up measures is minimal.

Consumers are used to seeing roll-ups in other realms (eg, 

 product reviews), but reporting roll-up measures of healthcare quality is relatively new. It is unclear whether consumers find these measures to be easier to understand than the component measures from which they are constructed or how these measures are interpreted. It is also unclear how viewing roll-up measures affects consumers’ choices of plans and providers, what consumers interpret rolled-up measures to signify, or whether roll-up measures simplify available data to the point where differences in performance no longer seem meaningful. Finally, little is known about whether roll-ups might deter crucial forms of consumer learning about performance metrics and provider variation in quality.

Consumer interpretation and use of roll-up measures likely vary depending on the level of aggregation of scores. The benefits and drawbacks of other stakeholders’ use of roll-up measures (eg, providers’ potential use of roll-ups for quality improvement purposes) were not documented here given the focus on report sponsors’ expectations about consumers’ use of roll-up scores; however, these expectations are important to understand. Because the reporting of roll-up measures is underway at federal agencies and among some report sponsors, it is critical to better understand how consumers perceive and respond to such measures. 

We found that report sponsors’ practices for calculating and reporting these measures are diverse. Report sponsors and national alliances that represent stakeholder groups affected by quality reporting feel that roll-up measures simplify large amounts of complex information for consumers, but run the risk of misinterpretation. The results of the study elucidate the need for research focused on the construction and reporting of roll-up measures, as well as consumer interpretations of roll-up measures.

 RAND Corporation (JLC, SCM, MLF, AMP, GM), Pittsburgh, PA; Severyn Group (LR), Ashburn, VA; University of Wisconsin (RG), Madison, WI; Yale University (MS), New Haven, CT; Shaller Consulting Group (DS), Stillwater, MN.

 This paper was supported by 2 cooperative agreements (2U18HS016980 and 1U18HS016978) from AHRQ to RAND Corporation and Yale University, respectively.

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (JLC, RG, GM, SCM, AMP, LR, DS, MS); acquisition of data (JLC, RG, GM, SCM, LR); analysis and interpretation of data (JLC, MLF, RG, GM, SCM, LR, DS, MS); drafting of the manuscript (JLC, MLF, AMP, MS); critical revision of the manuscript for important intellectual content (JLC, MLF, RG, GM, SCM, AMP, LR, DS); statistical analysis (JLC, MLF).

 Jennifer L. Cerully, PhD, RAND Corporation, 4570 Fifth Ave, Ste 600, Pittsburgh, PA 15213. E-mail: jcerully@rand.org.

1. Schlesinger M, Kanouse DE, Rybowski L, Martino SC, Shaller D. Consumer response to patient experience measures in complex information environments. 

. 2012;50(suppl):S56-S64. doi: 10.1097/MLR.0b013e31826c84e1.

This study identifies practices and perceptions around public reporting of â€œroll-upâ€

Grant Martsolf, PhD, MPH, RN

Articles

Primary Care Practice Structural Capabilities in Health Professional Shortage Areas

Using "Roll-up" Measures in Healthcare Quality Reports: Perspectives of Report Sponsors and National Alliances