At a glance:

• How well a newly minted doctor scores on their medical board exam appears linked to patients’ odds of dying or being readmitted to the hospital.
• Doctors’ performance on ratings of knowledge and skill taken periodically during residency training is not linked to patient outcomes.
• Findings offer reassurance that certification exams, which aim to demonstrate the competence of physicians, capture critical knowledge and clinical judgment skills for physicians.

How do we know whether newly minted doctors have what it takes to prevent patient deaths?

After completing residency training, graduating physicians typically take board certification exams at the time they enter practice — but surprisingly little is known about the ability of these standard tests to predict the things that count the most in a doctor’s performance, such as how likely their patients are to survive or to avoid a return trip to the hospital.

A new study, published May 6 in JAMA, found that internal medicine patients of newly trained physicians with top scores on the board certification exam — a comprehensive test usually taken after a physician completes residency training — had lower risk of dying within seven days of hospital admission or of being readmitted to the hospital.

The analysis was led by researchers at Harvard Medical School and the American Board of Internal Medicine (ABIM), the body that developed and regularly updates the exam that qualifies a physician as an internal medicine specialist. Some of the study authors, including lead author Bradley Gray, are employed by ABIM.

The findings, the team said, provide reassurance that the board exams in internal medicine are reflective of future physician performance on critical indicators of patient care and outcomes.

“These results confirm that certification exams are measuring knowledge that directly translates into improved outcomes for patients,” said study senior author Bruce Landon, professor of health care policy at HMS and an internal medicine doctor at Beth Israel Deaconess Medical Center.

Landon and colleagues said this is among the first known attempts to gauge the reliability of test scores in predicting patient outcomes.

The researchers also compared patient outcomes against “medical milestone” ratings developed by the Accreditation Council for Graduate Medical Education (ACGME). These ratings are based on periodic review of trainees’ knowledge and skills throughout the course of residency. The analysis found no link between patient outcomes and physicians’ scores on the milestone ratings.

Testing doctors in training

Anyone who’s watched a television medical drama knows that graduating from medical school is just the first step on a long journey of medical training and education. After students graduate from medical school, they go on to residency training in a specialty of their choice before taking their board certification exams and becoming fully qualified independent doctors in their particular area of medicine.

In recent decades, new ways have emerged to test the knowledge of these budding doctors.

In the field of internal medicine, resident competency is assessed in two ways. ACGME’s milestone ratings are administered at different times during residency training. The idea is that periodic milestone testing can give feedback to the physician in training on how well they are doing.

At the end of their training, whose length varies by specialty, almost all internal medicine residents take the ABIM’s certification exam, known as medical boards. This final comprehensive test gauges whether they are qualified to join the ranks of independently practicing doctors, capable of caring for patients without supervision.

Scientific approach to improving medical training

Most internal medicine specialists begin their careers as hospitalists, providing care to hospital inpatients. For the study, researchers analyzed patient outcomes of nearly 70,000 newly trained hospitalist physicians treating Medicare beneficiaries during 455,000 hospitalizations that took place from 2017 to 2019. The researchers compared outcomes for patients within the same hospitals who were cared for by doctors with different exam scores. This allowed the researchers to eliminate, or at least minimize, the effect of differences in patient populations, hospital resources, and other variations that might influence the odds of patient death or readmission, independent of a doctor’s performance.

Board exam performance was powerfully linked to patient risk of dying or hospital readmission. For example, there was an 8 percent reduction in the odds of dying within seven days of hospitalization in patients of physicians who scored in the top 25 percent on the exam, compared to the patients of physicians who scored in the bottom 25 percent on the exam, which was still a passing grade.

Even though physicians’ milestone ratings did not appear to predict patient outcomes, the researchers said that using them as a periodic assessment to help determine where a physician trainee may still be a valuable tool for people running training programs.

"This type of evidence-based assessment of our own testing tools provides valuable insights on which types of tests work for what purpose, which informs how they should be deployed in educating our future practitioners and leaders of medicine,” Landon said.

Authorship, funding, disclosures

Additional authors include Jennifer Stevens of HMS and Rebecca Lipner, Furman McDonald, and Jonathan Vandergrift of the American Board of Internal Medicine.

Gray, Lipner, McDonald, and Vandergrift reported that they are employees of ABIM. Landon reported receiving consulting fees from ABIM for ongoing work during the conduct of the study. No other disclosures were reported.