Currently Viewing:
The American Journal of Managed Care January 2014
Patient-Centered Medical Home Transformation With Payment Reform: Patient Experience Outcomes
Leonie Heyworth, MD, MPH; Asaf Bitton, MD, MPH; Stuart R. Lipsitz, ScD; Thad Schilling, MD, MPH; Gordon D. Schiff, MD; David W. Bates, MD, MSc; and Steven R. Simon, MD, MPH
Process of Care Compliance Is Associated With Fewer Diabetes Complications
Felicia J. Bayer, PhD; Deron Galusha, MS; Martin Slade, MPH; Isabella M. Chu, MPH; Oyebode Taiwo, MBBS, MPH; and Mark R. Cullen, MD
Evidence-Based Guidelines to Determine Follow-up Intervals: A Call for Action
Emilia Javorsky, MPH; Amanda Robinson, MD; and Alexa Boer Kimball, MD, MPH
Electronic Health Risk Assessment Adoption in an Integrated Healthcare System
Diana S. M. Buist, PhD, MPH; Nora Knight Ross, MA; Robert J. Reid, MD, PhD; and David C. Grossman, MD, MPH
Comorbidities and Cardiovascular Disease Risk in Older Breast Cancer Survivors
Reina Haque, PhD, MPH; Marianne Prout, MD; Ann M. Geiger, PhD; Aruna Kamineni, PhD; Soe Soe Thwin, PhD; Chantal Avila, MA; Rebecca A. Silliman, MD, PhD; Virginia Quinn, PhD; and Marianne Ulcickas Yood, DSc
The Effects of Federal Parity on Substance Use Disorder Treatment
Susan H. Busch, PhD; Andrew J. Epstein, PhD; Michael O. Harhay, MPH; David A. Fiellin, MD; Hyong Un, MD; Deane Leader Jr, DBA, MBA; and Colleen L. Barry, PhD, MPP
Currently Reading
Evaluation of Electronic Medical Record Administrative Data Linked Database (EMRALD)
Karen Tu, MD, MSc; Tezeta F. Mitiku, BSc, MSc; Noah M. Ivers, MD; Helen Guo, BSc, MSc; Hong Lu, PhD; Liisa Jaakkimainen, MD, MSc; Doug G. Kavanagh, BSEng, MD; Douglas S. Lee, MD, PhD; and Jack V. Tu, MD, PhD
Optimal Management of Diabetes Among Overweight and Obese Adults
Denison S. Ryan, MPH; Karen J. Coleman, PhD, MS; Jean M. Lawrence, ScD, MPH, MSSA; Teresa N. Harrison, SM; and Kristi Reynolds, PhD, MPH
Why Are Medicare and Commercial Insurance Spending Weakly Correlated?
Laurence C. Baker, PhD; M. Kate Bundorf, PhD; and Daniel P. Kessler, JD, PhD

Evaluation of Electronic Medical Record Administrative Data Linked Database (EMRALD)

Karen Tu, MD, MSc; Tezeta F. Mitiku, BSc, MSc; Noah M. Ivers, MD; Helen Guo, BSc, MSc; Hong Lu, PhD; Liisa Jaakkimainen, MD, MSc; Doug G. Kavanagh, BSEng, MD; Douglas S. Lee, MD, PhD; and Jack V. Tu, MD, PhD
The combination of electronic medical record data and administrative data provides the fullest picture of patient health histories.
Background: Primary care electronic medical records (EMRs) represent a potentially rich source of information for research and evaluation.

Objective: To assess the completeness of primary care EMR data compared with administrative data.

Study Design: Retrospective comparison of provincial health-related administrative databases and patient records for more than 50,000 patients of 54 physicians in 15 geographically distinct clinics in Ontario, Canada, contained in the Electronic Medical Record Administrative data Linked Database (EMRALD).

Methods: Physician billings, laboratory tests, medications, specialist consultation letters, and hospital discharges captured in EMRALD were compared with health-related administrative data in a universal access healthcare system.

Results: The mean (standard deviation [SD]) percentage of clinic primary care outpatient visits captured in EMRALD compared with administrative data was 94.4% (4.88%). Consultation letters from specialists for first consultations and for hospital discharges were captured at a mean (SD) rate of 72.7% (7.98%) and 58.5% (15.24%), respectively, within 30 days of the occurrence. The mean (SD)  capture within EMRALD of the most common laboratory tests billed and the most common drugs dispensed was 67.3% (21.46%) and 68.2% (8.32%), respectively, for all clinics.

Conclusions: We found reasonable capture of information within the EMR compared with administrative data, with the advantage in the EMR of having actual laboratory results, prescriptions for patients of all ages, and detailed clinical information. However, the combination of complete EMR records and administrative data is needed to provide a full comprehensive picture of patient health histories and processes, and outcomes of care.

Am J Manag Care. 2014;20(1):e15-e21
This study examined the comprehensiveness of primary care electronic medical record (EMR) data compared with administrative data.
  • Primary care EMRs have reasonably comprehensive capture of primary care physician visits, laboratory tests, and prescriptions by the EMR physician.
  • Laboratory tests and prescriptions by other providers may be missing; there is incomplete communication with specialists, and particularly poor communication between EMRs and hospitals.
  • Until integrated systems with open communication channels between all aspectsof the healthcare system are in place, the most comprehensive patient health histories can be achieved by combining information from complete EMR records and administrative data.
Although the United States and Canada have historically lagged behind other industrialized countries in the adoption of electronic medical records (EMRs) in primary care,1 with the introduction of the Health Information Technology for Economic and Clinical Health Act in 2009 in the United States2 and the establishment of Canada Health Infoway3 and provincial EMR adoption support programs in Canada, the uptake of EMRs in both countries is rapidly increasing.4 This development, especially in primary care physician practices, has resulted in a new, potentially rich source of clinical information not only for point-of-care clinical practice but also for secondary purposes such as research and quality performance evaluation.

Because of its single-payer healthcare system, Canada has comprehensive health-related administrative databases that cover the entire population. Complete, provincewide, population-level administrative databases have been shown to be highly accurate in capturing hospitalizations and prescriptions for Ontario residents 65 years and older.5 Also in these databases, physician billing data accurately capture frequency of patient encounters, but the depth and details of patient clinical encounters are unavailable. Indeed, a previous study in the United States found that EMR data in community health centers were more complete than Medicaid claims data for assessing diabetes preventive care.6

Because use of primary care EMR data in Canada for secondary purposes is in its relative infancy, we set out to determine the completeness and comprehensiveness of the EMR data compared with administrative data.


We developed the Electronic Medical Record Administrative data Linked Database (EMRALD) at the Institute for Clinical Evaluative Sciences (ICES). The Institute for Clinical Evaluative Sciences is a “prescribed entity” under provincial privacy legislation and thus is able to collect individual-level patient health information without patient consent based on policies and procedures in place to protect patient privacy and confidentiality.7 EMRALD was developed using data from family physicians (FPs) in Ontario using Practice Solutions EMR, the market-leading EMR software vendor in Ontario.8 All clinically relevant data are extracted from the participating physician’s EMR and each patient is anonymously linked, using their scrambled health card number, to the health-related administrative databases for the province of Ontario housed at ICES.

Physicians participate on a completely voluntary basis and are required to have had their EMR a minimum of 2 years to ensure that the EMR is adequately populated. Over the past decade Ontario has undergone major primary care reform such that the majority of all FPs in the province practice under one of the reform models that require “rostering” of patients (identification of patients with the 1 primary care provider who is most responsible for their care).9 All of the EMRALD FPs practice under one of the various primary care reform models of care. This analysis was confined to data that were captured in 2008 for rostered patients of participating FPs who had at least 1 visit to their physician in the 2 years before January 1, 2008.

Family Physician Clinic Visits

To determine the capture of all FP visits by patients rostered to EMRALD physicians (regardless of whether the visit was to an EMRALD physician), FP outpatient visits billed to the Ontario Health Insurance Plan (OHIP) physician billing database were compared with visits recorded in the EMR to determine the capture of FP visits by patients rostered to EMRALD physicians. Visit comparison between OHIP and EMRALD was confined to EMRALD physician OHIP visits only, to assess how completely EMRALD physicians were documenting patient visits within their EMR.

Specialist Consultations and Hospitalizations

To determine the capture of specialist visits and hospitalizations within EMRALD, OHIP billings by specialists performed in an office setting for an initial consultation and hospitalizations, as recorded in the Canadian Institute for Health Information Discharge Abstract Database, were compared with consultation letters and hospital discharge summaries in EMRALD within 14 days and within 30 days of the specialist billing date or the hospital discharge date.

Laboratory Data

A frequency count of OHIP laboratory fee codes identified the 20 most common laboratory tests ordered. Comparisons were made between the laboratory tests recorded on the same day in EMRALD and in OHIP.

Prescription Data

The EMR records when a drug is prescribed to the patient by the EMR physician, and the Ontario Drug Database (ODB) records when a drug was dispensed at a pharmacy (regardless of the prescriber). The EMR may also list drugs prescribed to a patient by other providers (ie, specialists), but only if such data are manually updated by the EMR-using physician; thus, the EMR is variably populated with these data. The prescription field within EMRALD and the prescriptions dispensed in ODB in 2008 were used to compile 2 drugs lists for the top 50 dispensed Canadian medications,10 grouped into 7 clinical categories. The 2 lists for each patient were compared. This analysis was confined to patients 65 years and older as of January 1, 2008, as the ODB only captures drugs universally for all Ontario residents 65 years and older.

Match Rates

Match rates were calculated for all parameters of comparison by calculating the mean match rate and standard deviation (SD) for all the clinics, giving each clinic equal weighting regardless of size.

This study received ethics approval from Sunnybrook Health Sciences Research Ethics Board.


Overall, there were 56,107 patients with an average age of 39.7 years from 54 physicians practicing in 15 geographically distinct clinics. The mean years since graduation of the physicians was 19.3 years (SD = 10.4 years). Seventy percent of the physicians were in urban practice, 56% were male, and 98% were in group practice. The average duration of time on the EMR was 4.5 years (SD = 2.6 years).

Family Physician Visits

Nearly 80% of the FP billings in OHIP that occurred in an office setting were captured in EMRALD. To determine whether the remaining billings were missing because patients were seen elsewhere (eg, a walk-in clinic) and therefore their visits were not recorded in the EMR, we limited the comparison to OHIP billings made by EMRALD physicians. In that case, we found that nearly all the billings for patients had a corresponding progress note entry in the EMR. Thus, we concluded that the approximately 15% of billings missing from the EMR were because patients were seen elsewhere and that the EMRALD participating physicians were fully using the EMR to record their patient encounters (Table 1).

Specialist Clinical Encounters and Hospitalizations

On average, just over two-thirds of initial specialist visits resulted in a consultation letter captured in EMRALD within 14 days of the visit; this percentage increased by less than 5% when the interval was expanded to 30 days (Table 1). Just over half of the hospital discharges had documentation of the hospitalization within EMRALD (Table 1).

Laboratory Tests and Prescriptions

We found that on average approximately three-fourths of the laboratory tests captured in EMRALD were also billed in OHIP, and most clinics had a higher match rate. When the laboratory tests recorded in OHIP were compared with what was recorded in EMRALD, approximately two-thirds of the tests billed to OHIP were captured (Table 1). Urinalysis tests had the poorest capture of all the laboratory tests in OHIP compared with EMRALD (Table 2).

Capture of prescriptions in EMRALD compared with drugs dispensed in ODB was high. However, only just over two-thirds of the drugs dispensed in ODB were captured in EMRALD (Table 1). Antibiotics (as represented by amoxicillin) were the drug class that had the poorest capture rate.  However, the capture rates for drugs to treat chronic conditions such as cardiovascular disease and endocrine diseases appeared to be relatively high (Table 3).


We found that primary care EMR data capture compared well with administrative data. Like previous US studies comparing EMR data with administrative data,6,11 our results support the idea that primary care process quality measures involving laboratory test ordering or prescriptions are better assessed through the EMR than through administrative data because the EMR is more likely to reflect what the primary care physician ordered or prescribed.

Although our results may seem to indicate that administrative data have superior capture of laboratory results, both administrative data and the EMR have their weaknesses. Unlike administrative data, the EMR has actual laboratory results available. However, the EMR often misses tests that were not ordered by the primary care provider. There was a larger SD for the lab results compared with the other assessed data elements because 2 clinics had much lower than average match rates. These 2 low-match-rate clinics both had far more  laboratory test data in the EMR than in the administrative data and were likely clinics where the patients used a hospital-based laboratory, which is not currently captured in provincial administrative databases.

Prescription data within the EMR contain prescriptions for patients of all ages (rather than just the elderly and/or those on social assistance, as in our provincial drug database), whereas the administrative data provide information on medications dispensed regardless of the prescriber. Because administrative data show the prescriptions dispensed while the EMR data shows the prescriptions prescribed, the match rates  should be considered in light of previous studies indicating that approximately 10% of  prescriptions do not get filled.12 The relatively low capture rate for antibiotics may reflect the use of walk-in clinics or the emergency department for the treatment of acute infectious conditions. An additional advantage of EMR data is the capture of prescribed medications that are over the counter or that are not on the provincial drug formulary.

We found that nearly three-fourths of specialist consultation letters were captured in the EMR, but hospital discharge summaries were less than optimally captured. These findings are similar to previous findings in the United States13 and highlight the need for improvements in communications between providers to ensure completeness of primary care EMRs.

Uncertainty regarding generalizability of our findings arises from the variability across provider users and across EMR systems.14 Each EMR software package has its own unique data structure. Until Canadian standards are developed, implemented, and enforced for entering and exporting EMR data elements, the quality of the data captured and the ability to identify discrete data elements in the EMR may not be the same across EMR software packages from different vendors. For instance, highly flexible, free-text-based EMR systems allow for multiple ways to enter data, requiring the identification of all options for denoting presence of disease conditions.

Copyright AJMC 2006-2019 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up