In primary care, nurse practitioners and physician assistants do not necessarily order more ancillary services, or more costly services among alternatives, than physicians.
Objectives: Debate continues on whether nurse practitioners (NPs) and physician assistants (PAs) are more likely to order ancillary services, or order more costly services among alternatives, than primary care physicians (PCPs). We compared prescription medication and diagnostic service orders associated with NP/PA versus PCP visits for management of neck or back (N/B) pain or acute respiratory infection (ARI).
Study Design: Retrospective, observational study of visits from January 2006 through March 2008 in the adult primary care practice of Kaiser Permanente in Atlanta, Georgia.
Methods: Data were obtained from electronic health records. NP/PA and PCP visits for N/B pain or ARI were propensity score matched on patient age, gender, and comorbidities.
Results: On propensity score-matched N/B pain visits (n = 6724), NP/PAs were less likely than PCPs to order a computed tomography (CT)/magnetic resonance image (MRI) scan (2.1% vs 3.3%, respectively) or narcotic analgesic (26.9% vs 28.5%) and more likely to order a nonnarcotic analgesic (13.5% vs 8.5%) or muscle relaxant (45.8% vs 42.5%) (all P ≤.05). On propensity score-matched ARI visits (n = 24,190), NP/PAs were more likely than PCPs to order any antibiotic medication (73.7% vs 65.8%), but less likely to order an x-ray (6.3% vs 8.6%), broad-spectrum antibiotic (41.5% vs 42.5%), or rapid strep test (6.3% vs 9.7%) (all P ≤.05).
Conclusions: In the multidisciplinary primary care practice of this health maintenance organization, NP/PAs attending visits for N/B pain or ARI were less likely than PCPs to order advanced diagnostic radiology imaging services, to prescribe narcotic analgesics, and/or to prescribe broad-spectrum antibiotics.
Am J Manag Care. 2017;23(4):225-231Takeaway Points
Practice variation by providers, rather than by provider type, may be more important in understanding differences in management of conditions commonly presented in primary care.
Nevertheless, concerns have been expressed that nurse practitioners (NPs) and physician assistants (PAs) might more frequently order ancillary services, or more costly services among alternatives, compared with primary care physicians. In this study of a group model health maintenance organization’s primary care practice, we found NPs and PAs were less likely to order:
Increasing the percentage of nurse practitioners (NPs) and physician assistants (PAs) in the primary care provider workforce has been suggested as one strategy for addressing the United States’ national shortage of primary care physicians (PCPs).1-5 Although NPs/PAs have clinical training and regulated scopes of practice that differ from those of PCPs, they are permitted to manage a range of medical conditions in ambulatory care.6-11 A recent survey of the US national population indicated broad acceptance of NPs/PAs as primary care providers.12
Currently, there is interest in how the addition of NPs/PAs to primary care might impact patient outcomes, medical service utilization, and costs.13-17 Prior studies have been relatively consistent in demonstrating that levels of patient satisfaction with care and quality of care are similar between NPs/PAs and physicians11,18-23; however, it is less clear whether care provided by NPs/PAs or PCPs affects medical services use and cost.
An NP/PA visit can be cost saving compared with a PCP visit because salary differentials lower the cost of time for patient evaluation and management.13,17,24,25 However, this cost advantage may be offset if NPs/PAs order ancillary services (ie, laboratory, radiology, pharmacy) at higher rates than PCPs, or order more costly services among alternatives (eg, computed tomography [CT] scan/magnetic resonance image [MRI] vs x-ray).
Studies show mixed results on use of ancillary services by provider type. One study found NPs/PAs ordered significantly more CTs/MRIs for primary care visits than physicians treating Medicare patients26; another found no difference in office-based care using the National Ambulatory Medical Care Survey (NAMCS).16 A study of Veterans Affairs patients with lower back pain found no statistically significant difference in clinical appropriateness of lumbar spine MRIs by provider type (physician, PA, or NP).27 In prescribing medications, rates of prescribing controlled medications28 and antibiotics16,29 were similar for NPs, PAs, and physicians.
The primary question of our retrospective observational study was: Do NPs/PAs attending visits for neck or back (N/B) pain or acute respiratory infection (ARI) in primary care order ancillary services at different rates than PCPs? We focused on these 2 medical conditions for several reasons. Musculoskeletal pain and respiratory infections are common reasons that adults present for medical care in the United States,30 and visits for N/B pain or ARI are frequently accompanied by orders for ancillary services. There are general concerns that, across all providers, certain types of ancillary services are overused and add cost without value—specifically, CTs/MRIs and narcotic analgesics in management of N/B pain,16,31,32 and antibiotics (particularly broad-spectrum antibiotics) in management of ARI.33-38
METHODSStudy Setting and Period
At the time of this study, Kaiser Permanente Georgia (KPGA) provided comprehensive medical services to approximately 240,000 enrollees per year (59% Caucasian, 33% African American) in the Atlanta area. The study protocol was reviewed, approved, and monitored by the KPGA Institutional Review Board.
During the 27-month study period (January 2006 through March 2008), approximately 180,000 KPGA members were empaneled to a PCP in the Adult Medicine department. The study period is limited to these 27 months because, beginning in 2006, an electronic health record (EHR) system was fully implemented—allowing for measurement of providers’ orders for medical services—and it ends in early 2008 when the multidisciplinary Adult Medicine department became a PCP-only department.39 More recent comparisons of practice variation are not available.
The sample used for analysis consisted of patients 18 years or older at the time of presentation for an “incident” N/B pain or ARI visit in the KPGA Adult Medicine department. An “incident” visit was considered to have occurred if the patient who presented for N/B pain or ARI had no visit in adult ambulatory medicine for N/B pain or ARI, respectively, for at least a period of 30 days prior to the visit. Using this definition, a single patient may have had multiple incident visits during the study period.
A visit for N/B pain or ARI was determined from specific International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis codes associated with the visit, either designated as a primary or secondary condition (eAppendix A [eAppendices available at ajmc.com]). Due to the broad range of ARIs, we selected specific ICD-9-CM diagnosis codes used in previously published studies as indicative of an ARI.33-38
We used propensity score matching of visits to reduce the effect of patient selection on comparisons of practice variation between NPs/PAs and PCPs when attending a visit for N/B pain or ARI. NPs/PAs tend to treat younger, healthier patients than physicians which, in turn, can affect rates and mix of ancillary service orders.11,16,40 Thus, we propensity score-matched NP/PA visits with PCP visits on preexisting patient characteristics to address potential selection issues.
Separate analyses were conducted for N/B pain and ARI visits. First, within each visit class, the propensity for a visit to be attended by a NP/PA
versus PCP was estimated with logistic regression.41-43 Second, based on the estimated propensity of NP/PA versus PCP attending a visit, each N/B pain or ARI visit attended by an NP/PA was matched with an N/B pain or ARI visit, respectively, attended by a PCP.
All analyses used computerized administrative and EHR databases. Records were linked by unique identifiers and sequenced by event dates. The EHR databases distinguished services ordered by the provider from services completed by the patient. We used services ordered rather than services delivered because rates of services delivered can be affected by factors such as patient access (eg, limited locations for CT/MRI services) or cost sharing (eg, co-payments for prescription medicines).44 Thus, services ordered are more representative of practice style than services delivered.
Services ordered at the time of the visit were linked to the incident N/B pain or ARI visit by unique visit numbers. All services ordered were attributed to the provider attending the visit. Thus, there is no attribution of orders by an NP/PA to a PCP providing practice supervision (as might occur in billing of “incidental to” visits).
We were advised by several senior physicians that some orders related to the incident visit might occur up to 3 to 5 days following conveyance to the attending provider of initial tests or consultation results. Thus, we considered a diagnostic service order or prescription order as related to the incident visit if it occurred within 5 days from the visit date and was ordered by a provider in the adult primary care department at the same primary care facility where the incident visit occurred. Follow-up orders occurred in less than 5% of NP/PA visits and/or PCP visits for either N/B pain or ARI (data available on request).
Dependent variables. The outcomes of interest were the percentages of visits associated with an order for a specific service class. The service order classes defined for N/B pain visits were: N/B x-rays, N/B CTs/MRIs, nonnarcotic analgesic prescriptions, narcotic analgesic prescriptions, and prescriptions for skeletal muscle relaxants. The service order classes defined for ARI visits were: ARI x-rays, ARI CTs/MRIs, rapid strep tests, any systemic antibiotic prescription, any broad-spectrum antibiotic prescription, and any prescription for relief of ARI symptoms (eg, decongestants, expectorants, respiratory system anti-inflammatory medications). We combined CTs with MRIs because preliminary review of the data indicated very low rates of MRIs relative to CTs, and availability of one or the other of these advanced diagnostic imaging services at selected facilities preferred by a patient might have affected a provider’s decision to schedule one or the other of these services.
Independent variable. The primary independent variable was the type of provider who attended the visit: NP/PA versus PCP.
Patient covariates. Covariates were: age at the time of the visit, gender, years of enrollment with KPGA at the time of the visit, and the presence (vs absence) of several major comorbidities (diabetes, hypertension, hyperlipidemia, cardiovascular disease [coronary artery disease, congestive heart failure, cerebrovascular disease], asthma or chronic obstructive pulmonary disease, or cancer) at the time of the visit.
The initial step in analysis was to compare the distribution of patient characteristics at presentation for a visit for N/B pain or ARI with respect to attending provider type: NP/PA versus PCP. Independence of patient characteristics from attending provider type (ie, NP/PA vs PCP) was assessed using a 2 test.
Propensity of a visit for N/B pain or ARI to have been attended by an NP/PA versus PCP was estimated using logistic regression including the patient covariates. This step resulted in a probability estimate of NP/PA versus PCP selection. Matching of 1 NP/PA visit with 1 PCP visit was done using a caliper of 0.25 times the standard deviation (SD); matching was done without replacement. Distributions of patient covariates before and after propensity score matching were compared using a 2 test. Before and after propensity score matching, the percentages of visits with a related diagnostic test or a medication order on an NP/PA versus PCP visit were compared using a 2 test.
Several sensitivity analyses were conducted. Results might be sensitive to the granularity of matching of NP/PA to PCP visits. Sensitivity of propensity score matching was tested with a finer caliper of 0.025 times the SD. Second, we examined percentages of visits with diagnostic radiology or medication orders by subgroups of visits classified according to the primacy of codes for presenting conditions, assuming that NPs/PAs or PCPs might use the EHR differently and, therefore, code visits differently according to sequence of “presenting” and “diagnosed” conditions. Finally, we estimated a logistic regression of each service order type as a function of NP/PA versus PCP using PROC GENMOD (SAS Institute, Cary, North Carolina) to evaluate whether clustering of service orders by provider might account for statistical significance of likelihood of a service type order by NP/PA or PCP.
All data management and statistical analyses were conducted using SAS version 9.4 (SAS Institute, Cary, North Carolina).
NPs/PAs attended to 16.2% of the visits (6724 of 41,404) with N/B pain as a primary or secondary diagnosis. NPs/PAs attended to 22% of the visits (24,190 of 109,844) with ARI as a primary or secondary diagnosis.
Table 1 displays patient characteristics on visits for N/B pain or ARI. Compared with PCPs, patients on N/B pain or ARI visits attended by NPs/PAs were younger, of shorter enrollment duration, and had lower prevalence of major comorbidities.
Table 2 displays the percent of N/B pain visits with orders for diagnostic radiology services and prescription medications by provider type, both before and after propensity score matching. After propensity score matching, PCPs were more likely to order an N/B-related CT/MRI (3.3% vs 2.1%; P <.01) or a narcotic analgesic (30.1% vs 26.9%; P <.01). On the other hand, NPs/PAs were more likely to order a nonnarcotic analgesic (13.5% vs 8.5%; P <.01) or a musculoskeletal relaxant (45.8% vs 42.5%; P <.01). N/B pain visits with an order for an x-ray did not differ between NPs/PAs or PCPs (21.6% vs 22.1%; P = .53).
Table 3 displays the percent of ARI visits with orders for diagnostic radiology services and prescription medications by provider type, both before and after propensity score matching. After propensity score matching, there were significant differences in practice between NPs/PAs and PCPs in treatment of ARI. Over all visits, PCPs were more likely to order ARI-related x-rays (8.6% vs 6.3%; P <.01), CTs/MRIs (0.5% vs 0.3%; P <.01), a broad-spectrum antibiotic (42.5% vs 41.5%; P = .03), or a rapid strep test (9.7% vs 6.3%; P <.01). NPs/PAs, however, were more likely to order any antibiotic (73.7% vs 65.8%; P <.01). As with N/B pain visits, this difference in practice variation between NPs/PAs and PCPs was the same as that observed before propensity score matching.
In the first sensitivity analysis, matching on a smaller caliper made no difference in the findings. Next, the practice pattern differences between NPs/PAs and PCPs over all visits on which a diagnosis of N/B pain (or ARI) was suggested were basically the same whether the relevant ICD-9-CM code was primary or secondary (Tables 2 and 3). Finally, there was some clustering by provider, and adjusting for this clustering tended to push the statistical significance of the NP/PA versus PCP effect toward the null (eAppendix B) in some cases; for example, CT/MRI for back pain pushing significance to nonsignificance (ie, P >.05). For most comparisons by provider type, however, the NP/PA versus PCP effect was significant whether the model did or did not account for clustering of service orders by provider.
The objective of our study was to ascertain whether NPs/PAs differed from PCPs in frequency of orders for diagnostic services or prescription medications when managing adults presenting with N/B pain or ARI in primary care. We used propensity score matching of visits to adjust for the fact that patients attended by NPs/PAs tended to be younger and have a lower prevalence of comorbidities, which could affect diagnostic or therapeutic treatment choice.
After matching, several important differences by type of primary care provider were noted in management of N/B pain: PCPs were more likely to order CTs/MRIs and narcotic analgesics and NPs/PAs were more likely to order nonnarcotic analgesics and muscle relaxants. Similarly, differences were noted in management of ARI: PCPs were more likely to order CTs/MRIs—although the rate of these orders was low—as well as x-rays, broad spectrum antibiotics, and rapid strep tests; NPs/PAs were more likely to order any antibiotic. Thus, on balance, PCPs tended to be more likely than NPs/PAs to order diagnostic or therapeutic services related to N/B pain and ARI visits and to order more costly services among alternatives (eg, CTs/MRIs vs x-rays for adults with N/B pain, broad spectrum antibiotics vs first-line general antibiotics for adults with ARIs).
Evidence from this health maintenance organization (HMO), therefore, differs from the results of other studies, suggesting that NPs/PAs might more frequently order diagnostic or therapeutic services for common conditions treated in primary care; or, among alternatives, order more costly services.27 Our study’s findings are, however, consistent with another recent study using data from the National Ambulatory Medical Care Survey (NAMCS), which found no significant differences between NPs/PAs and physicians in office-based practice when ordering “low-value” ancillary services.16 In our study, the pattern of ancillary services use suggests that NPs/PAs might have been more judicious in use of “low-value” ancillary services than PCPs. For management of back pain, overuse of CTs/MRIs and narcotic analgesics is a current concern.32 We found NPs/PAs had lower rates of use of CTs/MRIs and narcotic analgesics in management of N/B pain. In management of an ARI, overuse of antibiotics—particularly broad-spectrum antibiotics—is a long-standing concern.33-36,38 Overuse of rapid strep tests is another concern in management of ARIs,37 and we found NPs/PAs were less likely to order broad-spectrum antibiotics and rapid strep tests.
What factors might have contributed to this NPs/PA practice pattern? Training of NPs/PAs typically emphasizes patient education and self-management over other interventional strategies. Thus, NPs/PAs may be more comfortable in initially managing N/B pain or ARI with fewer ancillary services. It is also possible that NPs/PAs are more compliant than PCPs with clinical practice guidelines in management of N/B pain or ARI in primary care.
Sensitivity analyses suggest this study’s findings are robust. Matching a narrower caliper —one a tenth of that used for the findings discussed in this paper—yielded similar results. Frequencies of orders by NPs/PAs versus PCPs for visits related to N/B pain (or ARI) did not generally differ by whether N/B pain (or ARI) was indicated as a primary or secondary diagnosis. The clustering analyses do indicate some proportion in outcomes by provider type is due to practice variation among individual providers; however, the persistence of significance of the NP/PA effect after adjusting for provider clustering strongly suggests that practice variation by provider type is important.
Our study was conducted within the context of a single, group-model HMO in the southeastern United States. Because this HMO had a strong tradition encouraging multidisciplinary, collaborative primary care, study findings might not be generalizable to other settings with a different delivery model. NPs/PAs work under supervision of PCPs; however, we had no measure of how supervision practices might have influenced NP/PA ordering patterns. During the study period, NPs/PAs were relatively established in this HMO; their practice patterns might not represent practice patterns of newly hired NPs/PAs. This HMO had relatively well-defined practice guidelines for management of N/B pain and ARI. Rates of orders for medications reflect only orders for prescriptions and not over-the-counter medications. We did not investigate specific quality measures, so we cannot conclude that over- or underuse of specific diagnostic services or prescribed medications was beneficial or detrimental to patient health. The propensity score matching relied on a limited number of patient covariates, and does not necessarily account for illness acuity within the selected comorbidities. Other factors that varied across clinics where NPs/PAs practiced (eg, use of care managers in some clinics but not others) might also influence practice variation by provider type.
Other factors that we did not consider in our analyses could offset the potential savings in medical care delivery costs due to lower ancillary services rates on visits attended by NPs/PAs. Length of visit was not available, so we could not assess if longer NP/PA visits decreased visit productivity (in terms of visits per day) and attenuated labor cost savings due to lower NP/PA salaries.13,45 We did not examine variation by provider type in other utilization measures such as referrals or potentially avoidable hospital admissions. Other studies that have examined postvisit utilization generally find equal or lower rates of these classes of services following NP/PA visits compared with physician visits.15,22 Similarly, we show elsewhere that the extent of NP/PA integration into this HMO’s primary care delivery system did not increase levels of these broad classes of utilization across all medical conditions.39
In this group model HMO, NPs/PAs who attended visits related to N/B pain or ARI in adult primary care typically had lower rates of associated orders for diagnostic services or prescription medications than PCPs when treating patients of comparable age, gender, and comorbidities. Author Affiliations: School of Public Health, Georgia State University (DWR), Atlanta, GA; Center for Clinical and Outcomes Research, Kaiser Permanente (DWR, LFC, BER), Atlanta, GA; RAND Corporation (HL, MR, AM), Santa Monica, CA; Harvard University (HL, AM), Cambridge, MA; Massachusetts Health Policy Commission (DA), Boston, MA.
Source of Funding: Funds to conduct this study were provided by a grant from the American Academy of Family Physicians. The funding source had no role in the study design, data collection, interpretation of the results, and decision to submit the manuscript. Analyses and interpretations presented in this manuscript are solely those of the authors and do not represent the views of the sponsor or the authors’ employers.
Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.
Authorship Information: Concept and design (DWR, HL, MR, DA, AM); acquisition of data (DWR, LFC, DA); analysis and interpretation of data (DWR, HL, LFC, MR); drafting of the manuscript (DWR, MR); critical revision of the manuscript for important intellectual content (DWR, HL, MR, BER, AM); statistical analysis (DWR, LFC, MR); provision of patients or study materials (BER); obtaining funding (DWR, DA); administrative, technical, or logistic support (BER, AM); and supervision (DWR).
Address Correspondence to: Douglas W. Roblin, PhD, School of Public Health, Georgia State University, 1 Park Pl, Rm 662C, Atlanta, GA 30303. E-mail: DRoblin@gsu.edu.
1. Auerbach DI, Chen PG, Friedberg MW, et al. Nurse-managed health centers and patient-centered medical homes could mitigate expected primary care physician shortage. Health Aff (Millwood). 2013;32(11):1933-1941. doi: 10.1377/hlthaff.2013.0596.
2. Bodenheimer TS, Smith MD. Primary care: proposed solutions to the physician shortage without training more physicians. Health Aff (Millwood). 2013;32(11):1881-1886. doi: 10.1377/hlthaff.2013.0234.
3. Kirch DG, Henderson MK, Dill MJ. Physician workforce projections in an era of health care reform. Annu Rev Med. 2012;63:435-445. doi: 10.1146/annurev-med-050310-134634.
4. Kuo YF, Loresto FL Jr, Rounds LR, Goodwin JS. States with the least restrictive regulations experienced the largest increase in patients seen by nurse practitioners. Health Aff (Millwood). 2013:32(7):1236-1243. doi: 10.1377/hlthaff.2013.0072.
5. Sargen M, Hooker RS, Cooper RA. Gaps in the supply of physicians, advance practice nurses, and physician assistants. J Am Coll Surg. 2011;212(6):991-999. doi: 10.1016/j.jamcollsurg.2011.03.005.
6. Christian S, Dower C, O’Neil E. Overview of nurse practitioner scopes of practice in the United States—discussion. The Center for the Health Professions, University of California, San Francisco website. https://healthforce.ucsf.edu/publications/overview-nurse-practitioner-scopes-practice-united-states. Published December 1, 2007. Accessed June 23, 2016.
7. Dunker A, Krofah E, Isasi F. The role of physician assistants in health care delivery. National Governors Association website. https://www.nga.org/files/live/sites/NGA/files/pdf/2014/1409TheRoleOfPhysicianAssistants.pdf. Published September 22, 2014. Accessed June 23, 2016.
8. Gadbois EA, Miller EA, Tyler D, Intrator O. Trends in state regulation of nurse practitioners and physician assistants, 2001 to 2010. Med Care Res Rev. 2015;72(2):200-219. doi: 10.1177/1077558714563763.
9. Schiff M. The role of nurse practitioners in meeting increasing demand for primary care. National Governors Association website. https://www.nga.org/cms/home/nga-center-for-best-practices/center-publications/page-health-publications/col2-content/main-content-list/the-role-of-nurse-practitioners.html. Published December 20, 2012. Accessed June 23, 2016.
10. Morgan PA, Abbott DH, McNeil RB, Fisher DA. Characteristics of primary care office visits to nurse practitioners, physician assistants and physicians in United States Veterans Health Administration facilities, 2005 to 2010: a retrospective cross-sectional analysis. Hum Resour Health. 2012;10:42. doi: 10.1186/1478-4491-10-42.
11. Roblin DW, Becker ER, Adams EK, Howard DH, Roberts MH. Patient satisfaction with primary care: does type of practitioner matter? Med Care. 2004;42(6):579-590.
12. Dill MJ, Pankow S, Erikson C, Shipman S. Survey shows consumers open to a greater role for physician assistants and nurse practitioners. Health Aff (Millwood). 2013;32(6):1135-1142. doi:
13. Basu S, Landon BE, Song Z, Bitton A, Phillips RS. Implications of workforce and financing changes for primary care practice utilization, revenue, and cost: a generalizable mathematical model
for practice management. Med Care. 2015;53(2):125-132. doi: 10.1097/MLR.0000000000000278.
14. Hughes DR, Jiang M, Duszak R Jr. A comparison of diagnostic imaging ordering patterns between advanced practice clinicians and primary care physicians following office-based evaluation and management visits. JAMA Intern Med. 2015;171(1):101-107. doi: 10.1001/jamainternmed.2014.6349.
15. Kuo YF, Chen NW, Baillargeon J, Raji MA, Goodwin JS. Potentially preventable hospitalizations in Medicare patients with diabetes: a comparison of primary care provided by nurse practitioners versus physicians. Med Care. 2015;53(9):776-783. doi: 10.1097/MLR.0000000000000406.
16. Mafi JN, Wee CC, Davis RB, Landon BE. Comparing use of low-value health care services among U.S. advanced practice clinicians and physicians. Ann Intern Med. 2016;165(4):237-244. doi: 10.7326/M15-2152.
17. Perloff J, DesRoches CM, Buerhaus P. Comparing the cost of care provided to Medicare beneficiaries assigned to primary care nurse practitioners and physicians. Health Serv Res. 2016;51(4):1407-1423. doi: 10.1111/1475-6773.12425.
18. Everett C, Thorpe C, Palta M, Carayon P, Bartels C, Smith MA. Physician assistants and nurse practitioners perform effective roles on teams caring for Medicare patients with diabetes. Health Aff (Millwood). 2013;32(11):1942-1948. doi: 10.1377/hlthaff.2013.0506.
19. Horrocks S, Anderson E, Salisbury C. Systematic review of whether nurse practitioners working in primary care can provide equivalent care to doctors. BMJ. 2002;324(7341):819-823.
20. Laurant M, Harmsen M, Wollersheim H, Grol R, Faber M, Sibbald B. The impact of nonphysician clinicians: do they improve the quality and cost-effectiveness of health care services? Med Care Res Rev. 2009;66(suppl 6):36S-89S. doi: 10.1177/1077558709346277.
21. Lenz ER, Mundinger MO, Kane RL, Hopkins SC, Lin SX. Primary care outcomes in patients treated by nurse practitioners or physicians: two-year follow-up. Med Care Res Rev. 2004;61(3):332-351.
22. Mundinger MO, Kane RL, Lenz ER, et al. Primary care outcomes in patients treated by nurse practitioners or physicians: a randomized trial. JAMA. 2000;283(1):59-68.
23. Obman-Strickland PA, Orzano AJ, Hudson SV, et al. Quality of diabetes care in family medicine practices: influence of nurse-practitioners and physician’s assistants. Ann Fam Med. 2008;6(1):14-22. doi: 10.1370/afm.758.
24. Essary AC, Green EP, Gans DN. Compensation and production in family medicine by practice ownership. Health Serv Res Manag Epidemiol. 2016;3:1-5. doi: 10.1177/2333392815624111.
25. Roblin DW, Howard DH, Becker ER, Adams EK, Roberts MH. Use of midlevel practitioners to achieve labor cost savings in the primary care practice of an MCO. Health Serv Res. 2004;39(3):607-626.
26. Hughes DR, Jiang M, Duszak R Jr. A comparison of diagnostic imaging ordering patterns between advanced practice clinicians and primary care physicians following office-based evaluation and management visits. JAMA Intern Med. 2015;171(1):101-107. doi: 10.1001/jamainternmed.2014.6349.
27. Gidwani R, Sinnott P, Avoundjian T, Lo J, Asch SM, Barnett PG. Inappropriate ordering of lumbar spine magnetic resonance imaging: are providers choosing wisely? Am J Manag Care. 2016:22(2):e68-e76.
28. Cipher DJ, Hooker RS, Guerra P. Prescribing trends by nurse practitioners and physician assistants in the United States. J Am Acad Nurse Pract. 2006;18(6):291-296.
29. Ladd E. The use of antibiotics for viral upper respiratory tract infections: an analysis of nurse practitioner and physician prescribing practices in ambulatory care, 1997-2001. J Am Acad Nurse Pract. 2005;17(10):416-424.
30. National Ambulatory Medical Care Survey: 2010 summary tables. CDC National Center for Health Statistics website. http://www.cdc.gov/nchs/data/ahcd/namcs_summary/2010_namcs_web_tables.pdf. Accessed October 2, 2015.
31. Daubresse M, Chang HY, Yu Y, et al. Ambulatory diagnosis and treatment of non-malignant pain in the United States, 2000-2010. Med Care. 2013;51(10):870-878. doi: 10.1097/MLR.0b013e3182a95d86.
32. Mafi JN, McCarthy EP, Davis RB, Landon BE. Worsening trends in the management and treatment of back pain. JAMA Intern Med. 2013;173(17):1573-1581. doi: 10.1001/jamainternmed.2013.8992.
33. Barnett ML, Linder JA. Antibiotic prescribing for adults with acute bronchitis in the United States, 1996-2010. JAMA. 2014;311(19):2020-2022. doi: 10.1001/jama.2013.286141.
34. Barnett ML, Linder JA. Antibiotic prescribing to adults with sore throat in the United States, 1997-2010. JAMA Intern Med. 2014;174(1):138-140. doi: 10.1001/jamainternmed.2013.11673.
35. Grijalva CG, Nuorti JP, Griffin MR. Antibiotic prescription rates for acute respiratory tract infections in US ambulatory care settings. JAMA. 2009;302(7):758-766. doi: 10.1001/jama.2009.1163.
36. Lee GC, Reveles KR, Attridge RT, et al. Outpatient antibiotic prescribing in the United States: 2000 to 2010. BMC Med. 2014;12:96. doi: 10.1186/1741-7015-12-96.
37. Nakhoul GN, Hickner J. Management of adults with acute streptococcal pharyngitis: minimal value for backup strep testing and overuse of antibiotics. J Gen Intern Med. 2013;28(6):830-834. doi: 10.1007/s11606-012-2245-8.
38. Steinman MA, Landefeld CS, Gonzales R. Predictors of broad-spectrum antibiotic prescribing for acute respiratory tract infections in adult primary care. JAMA. 2003;289(6):719-725.
39. Liu H, Robbins M, Mehrotra A, et al. The impact of using mid-level providers in face-to-face primary care on health care utilization. Med Care. 2017;55(1):12-18.
40. Halter M, Drennan V, Chattopadhyay K, et al. The contribution of physician assistants in primary care: a systematic review. BMC Health Serv Res. 2013;13:223. doi: 10.1186/1472-6963-13-223.
41. Austin PC, Tu JV. Bootstrap methods for developing predictive models in cardiovascular research. Am Stat. 2004;58(2):131-137. doi: 10.1198/0003130043277.
42. D’Agostino RB Jr. Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat Med. 1998;17(19):2265-2281.
43. Rubin DB. Estimating causal effects from large data sets using propensity scores. Ann Intern Med. 1997;127(8 pt 2):757-763.
44. Liang S-Y, Eaton L, Pu J, Chen T, Luft H. Variations in prescription fills in an insured population. Poster presented at: AcademyHealth, Annual Research Meeting; June 15, 2015; Minneapolis, MN. https://academyhealth.confex.com/academyhealth/2015arm/meetingapp.cgi/Session/1814. Accessed June 23, 2016.
45. Larson EH, Hart LG, Ballweg R. National estimates of physician assistant productivity. J Allied Health. 2001;30(3):146-152.