The Coalition for Physician Accountability's Undergraduate Medical

UME-GME REVIEW COMMITTEE

The Coalition for

Physician Accountability’s

Undergraduate Medical

Education-Graduate

Medical Education

Review Committee (UGRC):

Recommendations for Comprehensive

Improvement of the UME-GME Transition

https://physicianaccountability.org/

UME-GME REVIEW COMMITTEE

Table of ContentsTable of Contents

Competencies

and Assessments

FIVE RECOMMENDATIONS

Competencies

and Assessments

FIVE RECOMMENDATIONS

UGRC Members

Competencies

and Assessments

FIVE RECOMMENDATIONS

Competencies

and Assessments

FIVE RECOMMENDATIONS

Competencies

and Assessments

FIVE RECOMMENDATIONS

Competencies

and Assessments

FIVE RECOMMENDATIONS

Competencies

and Assessments

FIVE RECOMMENDATIONS

Competencies

and Assessments

FIVE RECOMMENDATIONS

Competencies

and Assessments

FIVE RECOMMENDATIONS

Acknowledgments

Final UGRC

Recommendations

and Themes

10-26

Executive Summary

6-9

UGRC Process

28-33

References

Limitations

Overview

Learner’s Journey

Future Ideal State

34-37

Impact of Public

Commentary

Appendix A:

Glossary of Terms

and Abbreviations

45-46

Appendix B:

Workgroup Ishikawa Diagrams (Fishbones)

Created for Root Cause Analysis

47-50

Appendix C:

Final Recommendations

with Complete Templates

50-123

Appendix D:

Preliminary Recommendations

Released on April 26, 2021

125-138

Appendix E:

Analysis of the Public

Comments

139-275

Consolidation and

Sequencing

39-41

UME-GME REVIEW COMMITTEE

Overview

In 2020, the Coalition for Physician Accountability (Coalition) formed a new committee to examine the transition

from undergraduate medical education (UME) to graduate medical education (GME). The UME-GME Review

Committee (the “UGRC” or the “Committee”) was charged with the task of recommending solutions to identied

challenges in the transition. These challenges are well known, but the complex nature of the transition together

with the reality that no single entity has responsibility over the entire ecosystem has perpetuated the problems

and thwarted attempts at reform.

Using deliberate and thoughtful methods, the UGRC spent 10 months exploring, unpacking, discussing, and

debating all aspects of the UME-GME transition. The Committee envisioned a future ideal state, performed a root-

cause analysis of the identied challenges, repeatedly sought stakeholder input, explored the literature, sought

innovations being piloted across the country, and generated a preliminary set of potential solutions to the myriad

problems associated with the transition. Initial recommendations were widely released in April 2021, and feedback

was obtained from organizational members of the Coalition as well as interested stakeholders through a public

call for comment. This feedback was instrumental to rening, altering, and improving the recommendations into

their nal form. The UGRC also responded to feedback by consolidating similar recommendations, organizing

them into more cogent themes, and sequencing them to guide implementation.

The UGRC has presented a total of 34 nal recommendations, organized around nine themes, for comprehensive

improvement of the UME-GME transition. The Committee has formally handed o these recommendations to the

Coalition for their consideration and implementation. Importantly, the UGRC strove to abide by an agreed upon set

of guiding principles that gave primacy to the public good and that championed diversity, equity, and inclusion. The

Committee believes that the recommendations are interconnected and should be implemented as a complete

set. Doing so will create better organizational alignment, likely decrease student costs, reduce work, enhance

wellness, address inequities, better prepare new physicians, and enhance patient care.

UME-GME REVIEW COMMITTEE

UGRC Members

The members of the UME-GME Review Committee (UGRC), their pertinent constituency or organizational

aliation, and their workgroup assignments are listed in the table below. Please refer to the section of this report

on “UGRC Process” for details about each workgroup. A complete glossary of terms and constituency/organization

names can be found in Appendix A.

Name Constituency/Organization Workgroup Assign-ment

Richard Alweis DIO B

Steven Angus DIO A

Michael Barone NBME A

Jessica Bienstock DIO D and Bundling

Maura Biszewski AOA D

Craig Brater ECFMG A and Bundling

Jesse Burk Rafel Resident C

Andrea Ciccone Lead Coalition Sta Member Unassigned

Susan Enright (Workgroup B Leader) Medical Education B

Sylvia Guerra Student B and DEI

Daniel Giang (Bundling Workgroup Leader) DIO C and Bundling

John Gimpel NBOME B

Karen Hauer (Workgroup A Leader) Medical Education A

Carmen Hooker Odom Public Member B

Donna Lamb NRMP B

Grant Lin Resident D, DEI, and Bundling

Elise Lovell (UGRC co-chair) OPDA C

George Mejicano (UGRC co-chair) Medical Education D and DEI

Thomas Mohr AACOM C

Greg Ogrinc ABMS D and DEI

Juhee Patel Student A

Michelle Roett (DEI Workgroup Leader) Medical Education D and DEI

Dan Sepdham Residency Program Director C

Susan Skochelak AMA D

Julie Story Byerley (Workgroup D Leader) Medical Education D and Bundling

Jennifer Swails (Workgroup C Leader) Residency Program Director C and Bundling

Jacquelyn Turner Clerkship Director C and DEI

Alison Whelan AAMC B

Pamela Williams Medical Education A

William Wilson Public Member A

UME-GME REVIEW COMMITTEE

Acknowledgements

The UGRC would like to acknowledge the signicant contributions of the following individuals, whose eorts were

integral to the successful completion of the Committee’s charge.

We greatly appreciate the dedication and commitment demonstrated over the past year by the UGRC

workgroup leads: Julie Byerley, Susan Enright, Daniel Giang, Karen Hauer, Michelle Roett, and Jennifer Swails. Our

public members, Carmen Hooker Odom and Reverend William Wilson, consistently focused our work on our

ultimate responsibility, the public good.

Andrea Ciccone (NBME, AOA), served as primary Coalition sta support to the UGRC, and provided invaluable

insights, perspectives, and organization.

We sincerely thank our outstanding project manager Chris Hanley (AAMC) and communications director Joe

Knickrehm (FSMB). Research librarians Kris Alpi, Robin Champieux, and Andrew Hamilton (Oregon Health & Science

University) enthusiastically guided our evidence informed approach. A team consisting of Dana Kerr, Matthew

Roumaya, Carol Morrison, Ulana Dubas, and Lauren Foster (NBME) performed the important analysis of the public

commentary, while Susan Morris and Sheila FitzPatrick (ABMS) managed graphic design. Our medical writer

Victoria Stagg Elliott (AMA) contributed signicantly to the creation of this report.

The members of the Coalition Management Committee met throughout this process and oered relevant

guidance and context.

Finally, we are eternally grateful to each member of the UGRC, who contributed their time, passion, expertise, and

experience from across the arc of medical education, in the common cause of improving the UME-GME transition

for all involved, and improving the medical care provided in our society.

Elise Lovell, MD

George Mejicano, MD, MS

UGRC co-chairs

UME-GME REVIEW COMMITTEE

Executive Summary

In the summer of 2020, a Planning Committee of the Coalition for Physician

Accountability (Coalition) selected the members of a new committee – the

Undergraduate Medical Education (UME) to Graduate Medical Education

(GME) Review Committee (UGRC) – and charged them with the task of

recommending solutions to identied challenges in the UME-GME transition.

The UGRC is pleased to submit this report, which includes the 34 nal

recommendations for comprehensive improvement of the UME-GME

transition, to the Coalition for their consideration and implementation.

Introduction:

The charge to the UGRC stated that there are identied challenges in the transition between medical school and

residency that are negatively impacting the UME-GME transition.1 These challenges include, but are not limited to,

the following:

• Disproportionate attention towards nding and lling residency positions rather than on assuring learner

competence and readiness for residency training;

• Unacceptable levels of stress on learners and program directors throughout the entire process;

• Inattention to optimizing congruence between the goals of the applicants and the mission of the programs to

ensure the highest quality health care for patients and communities;

• Mistrust between medical school ocials and residency program personnel;

• Overreliance on licensure examination scores in the absence of valid, trustworthy measures of students’

competence and clinical abilities;

• Lack of transparency to students on how residency selection actually occurs;

• Increasing nancial costs to students as well as opportunity costs to programs associated with skyrocketing

application numbers;

• The presence of individual and systemic bias throughout the transition; and

• Inequities related to specic types of applicants such as international medical graduates.

In recent years, these and related challenges have expanded to the point that they are causing severe strain on

the entire system. Simply put, there is an emerging consensus and urgency to bring forth solutions and as stated

by the Planning Committee,1 that the “UME-GME community is energized at this moment to solve these problems,

and should therefore act boldly and fairly with transparency, while thoughtfully considering stakeholder input, and

utilizing data when available.”

In addition to understanding the challenges noted above, the UGRC had to develop a shared concept of what

comprises the “UME-GME transition.” Through its deliberations, the Committee came to a collective understanding

that the transition encompasses a complex ecosystem involving many individuals and organizations. The transition

begins during the preclinical phase of medical school as students consider specialty options, are counseled by

UME-GME REVIEW COMMITTEE

Executive Summary

mentors and faculty advisors, and embark on the long journey of professional identity formation. During their

clinical years, students participate in patient care in numerous settings and on dierent rotations, choose a

variety of electives, decide on a specialty, prepare application materials, research residency programs, apply to

many programs, are oered and partake in interviews, interact with program personnel, are selected through a

matching process, undergo hiring and credentialing, complete advanced skills training courses, experience major

life transitions, initiate new support structures, begin employment, participate in orientation, assume signicantly

more patient care responsibilities, and embed themselves within a learning and work environment that they will

call home for the next three to seven years. In other words, the UME-GME transition is not simply the application,

interview, and match process. Moreover, the transition does not end at the start of orientation to their rst year of

training. For unmatched students and international medical graduates, the process may take even longer.

As learners navigate through the UME-GME transition, they interact with numerous organizations with jurisdiction

over specic components of the process. Each organization plays a role and impacts the success of the

transition. However, the ecosystem is not governed by a single entity. In essence, it is a decentralized collection of

interdependent parts, each with their own interests, which currently do not communicate eectively or function

cohesively. Solutions that bring the components of the transition into better alignment could have many positive

outcomes and will likely decrease student costs, reduce work, enhance wellness, address inequities, better

prepare new physicians, and enhance patient care.

Background:

In 2018, a national conversation culminated regarding the use of numeric scores associated with medical licensing

examinations in residency applicant screening and selection. In response, the chief executive ocers of ve

national organizations (AMA, AAMC, ECFMG, FSMB, and NBME) agreed to co-sponsor an Invitational Conference on

USMLE Scoring (InCUS).

InCUS took place in March 2019 with a primary goal of reviewing the practice of numeric

score reporting. Three of the recommendations that emerged focused on the USMLE:

(a) Reduce the adverse impact of the overemphasis on USMLE performance in residency screening and

selection through consideration of changes such as pass/fail scoring;

(b) Accelerate research on the correlation of USMLE performance to measures of residency performance

and clinical practice; and

In contrast, the fourth InCUS recommendation focused on the UME-GME transition: Convene a cross-organizational

panel to create solutions for the assessment and transition challenges from UME to GME. The nal report from

InCUS noted that there was general agreement that changes in scoring of licensure examinations would not

address important aspects of the UME-GME transition system that needed attention. “It was acknowledged

that many organizations and stakeholder groups have responsibility for improving this transition. Yet if many are

responsible, a concern exists that no one group will take ownership or feel empowered to carry on the broader

conversation necessary to bring about appropriate change.”

In September 2019, a proposal was made to the Coalition to convene a UME-GME Review Committee in line

with the fourth recommendation from InCUS.

As a result, a Planning Committee was created by the Coalition

to develop the construct, membership, and charge of the Review Committee, which would be responsible for

recommending solutions to identied challenges in the UME-GME transition.

UME-GME REVIEW COMMITTEE

Executive Summary

The UGRC’s Guiding Principles

As stated above, the UGRC was charged with the task of recommending solutions to identied challenges in the

UME-GME transition. Although the Committee was encouraged to act boldly, thoughtfully consider stakeholder

input, and utilize data whenever possible, the UGRC’s primary goals were to ensure learner competence and

readiness for residency and to foster wellness in learners, sta, faculty members, and program directors.

addition, the UGRC was tasked to devote attention to the following items:

• Optimizing t between applicants and programs to ensure the highest quality health care for patients

and communities;

• Increasing trust between medical schools and residency programs;

• Mitigating current reliance on licensure examinations in the absence of valid, standardized, trustworthy

measures of students’ competence and clinical care;

• Increasing transparency for applicants to understand how residency selection operates;

• Considering the needs of all types of applicants in making its recommendations;

• Considering nancial cost to applicants throughout the application process; and

• Minimizing individual and systemic bias throughout the UME-GME transition process.

The UGRC melded these principles into a single tenet that was kept front of mind during its deliberations and

related work: above all else, the UME-GME transition must optimally serve the public good. Inherent to that tenet,

the Committee consistently focused on the importance of increasing diversity, enhancing equity, and championing

inclusion.

The Work of the UGRC

Seven consensus steps prepared the UGRC to successfully accomplish the task of generating recommendations:

• Elaborate the charge to include optimal preparation for residency by leveraging learners’ time and

experiences between the Match and the initial months of training.

• Require level setting to ensure that all UGRC members had common understanding of the UME-GME

transition.

• Use the concept of backward design to envision a future ideal state that helps create a system that

produces it.

• Produce Ishikawa diagrams (i.e., shbones) to determine the root causes that underly the many

challenges currently associated with the transition.

• Articulate the desired outcome and understand the root problems before generating solutions.

• Identify potential solutions and innovations described in the literature or implemented by institutions

across the country.

• Embrace a consensus approach to endorsing recommendations, informed by available evidence.

Generation and Adoption of Preliminary Recommendations

The UGRC did not begin the process of generating potential solutions to the identied problems of the transition

until the work described above was complete. Even then, the generation of the preliminary recommendations

was focused and deliberate to ensure that background material could be assembled, that each potential solution

UME-GME REVIEW COMMITTEE

Executive Summary

was thoughtfully considered, and that there was ample time and space to discuss contentious ideas. Each

recommendation was linked to the future ideal state as well as to root causes of problems with the transition.

In addition, the co-chairs decided to frame each recommendation in broad terms, to include specic

examples on how a recommendation might be implemented, and to list both pros and cons for each potential

recommendation. Successful implementation of the UGRC’s recommendations relies on the cooperation of

multiple entities since the challenges within the transition are interdependent and not under the control of any

one organization or stakeholder group. Recommendations based on principles and that describe characteristics

of what can be achieved are more likely to garner support compared to granular recommendations that might

be readily dismissed as unrealistic or politically dicult. In addition, recommendations with a high degree of

consensus will be harder to ignore than those adopted by the UGRC by a simple majority.

In April 2021, the UGRC adopted 42 preliminary recommendations organized around 12 themes. The preliminary

recommendations and pertinent background material were presented to the Coalition, followed one week later

by their widespread release and a call for public comment.

Response to Feedback and Next Steps

Feedback about the preliminary recommendations was obtained from the organizational members of the

Coalition as well as from stakeholders through a public, month-long call for comments. This feedback was shared

with each member of the UGRC so that input from the Coalition and external stakeholders could inform the

Committee’s nal recommendations. Feedback obtained by the UGRC co-chairs through dialogue with students,

program directors, DIOs, medical educators, medical school deans, and international medical graduates –

obtained through purposeful outreach to those groups – was considered before nalizing the recommendations.

In response to all feedback, the UGRC made important changes to its preliminary recommendations. The changes

included signicant editing, clarication, and renement of language; complete reworking of a recommendation

addressing application inflation; and judiciously combining similar ideas to reduce the overall number of

recommendations. Of note, 32 of the preliminary recommendations were impacted by the feedback obtained

through public commentary.

Further, the co-chairs created a “bundling workgroup” tasked to consolidate similar recommendations, to

sequence those that were interdependent with one another, and to re-organize them into more cogent themes.

As a result of these eorts, the UGRC has adopted 34 nal recommendations, organized around nine themes,

to comprehensively improve the UME-GME transition. Moreover, the recommendations within each theme are

sequenced in chronologic order to guide their implementation. A fully textualized and comprehensive narrative on

each recommendation can be found in Appendix C.

The Committee believes that each proposed change will produce positive results and that implementation of

the complete set of recommendations will improve the entire transition. The UGRC also recognizes that each

recommendation may be categorized as transactional, investigational, or transformational in nature. Though

certain recommendations are designed to garner “early wins” by reducing the signicant stress felt by students

and program directors, the UGRC believes that the transformational recommendations are of greatest

importance because they align the medical community with a shared interest to promote the public good.

With the delivery of the 34 nal recommendations and this accompanying report, the work of the UGRC is now

complete. The Coalition will meet in late summer 2021 to discuss the nal recommendations and consider next

steps towards implementation.

UME-GME REVIEW COMMITTEE

Collaboration and Continuous

Quality Improvement

RECOMMENDATIONS

Convene a national ongoing committee to manage continuous quality improvement

of the entire process of the UME-GME transition, including an evaluation of the

intended and unintended impact of implemented recommendations.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

One of the challenges in creating alignment and making improvements is the lack of a single body with

broad perspective over the entire continuum. This creates a situation where organizations and institutions

are unnecessarily and counterproductively isolated, without a shared mental model or mission. A convened

committee, that includes learner and public representatives, should champion continuous improvement to the

UME-GME transition, with the focus on the public good.

THEME

In addition to supporting collaboration around the UME-GME transition, this national

committee should: develop and articulate consensus around the components of

a successful residency selection cycle; explore the growing number of unmatched

physicians in the context of a national physician shortage; and foster future research

to understand which factors are most likely to translate into physicians who fulll the

physician workforce needs of the public.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Currently, the medical education community lacks a shared mental model of what constitutes a successful

transition from UME to GME, and also what factors predict that success. The lack of agreement leads to conflict

over the content of applications as well as the resources required for a residency selection cycle. Success could

include simple educational outcomes such as completing training, board certication, or lack of remediation.

Alternatively, applicant-specic factors may be more important, such as likelihood of choosing the same

program again. Success may be dened solely on the public good, based on the ll rate of programs and

the number of physicians practicing in underserved areas. Or, it may be that successful residency selection is

institutionally specic based on its mission and community served, with some institutions focused on research

and others on rural communities. The committee should articulate the factors associated with a successful

residency selection cycle so they can be appropriately emphasized in the UME-GME transition, especially as

changes are made to the process.

The committee should report on data trends, implications, and recommended interventions to address the

growing number of unmatched physicians. This analysis should include demographic data to examine diversity,

specialty disparities in unmatched students, number of applications, grading systems, participation in SOAP,

post-SOAP unmatched candidates, match rate in subsequent years of re-entering the match pool, and attrition

rates of learners during residency. This recommendation is intended to urge UME programs and institutions

to utilize a continuous quality improvement approach and review unmatched graduates by specialties,

demographics, number of programs applied to, and clinical grading; to oer alternative pathways; and to add

The UGRC recommends the following, organized around nine themes:

UME-GME REVIEW COMMITTEE

Collaboration and Continuous Quality Improvement

RECOMMENDATIONS

faculty development for clinical advising. Both UME and GME data would identify patterns within the continuum

of medical education that negatively impact unmatched physicians and attrition rates of GME programs. Ideally,

shared resources and innovation across the continuum would be identied and disseminated.

Graduates of U.S. medical schools ll many residency positions, which means GME is constrained by the

decisions made by U.S. medical school admissions committees. However, international medical graduates are

also considered at many programs and provide an opportunity to serve the public good. The committee should

foster research to help program directors understand which applicant characteristics are useful indicators to

address ongoing medical workforce issues. Further changes to the transition should be informed by evidence

whenever possible.

The U.S. Centers for Medicare and Medicaid Services (CMS) should change the

current GME funding structure so that the Initial Residency Period (IRP) is calculated

starting with the second year of postgraduate training. This will allow career choice

reconsideration, leading to improved resident wellbeing and positive eects on the

physician workforce.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Given the timing of the residency recruiting season and the Match, students have limited time to denitively

establish their specialty choice. If a resident decides to switch to another program or specialty after beginning

training, the hospital may not receive full funding due to the IRP and thus be far less likely to approve such a

change. The knowledge that residents usually only have one chance to choose a specialty path increases the

pressure on the entire UME-GME transition. Furthermore, educational innovation is limited without flexibility for

time-variable training.

UME-GME REVIEW COMMITTEE

Diversity, Equity, and Inclusion

RECOMMENDATIONS

Specialty-specic salutary practices for recruitment to increase diversity across

the educational continuum should be developed and disseminated to program

directors, residency programs, and institutions.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Recognizing that program directors, residency programs, and institutions have wide variability in goals, denitions,

and community needs for increasing diversity, shared resources should be made available for mission-aligned

entities, with specialty-specic contributions including successful strategies and ongoing challenges. This

recommendation is intended for specialty organizations to perform workforce evaluations and specically

address diversity, equity, and inclusion (DEI) associated with specialty-specic disparities in recruitment.

THEME

Members of the medical educational continuum must receive continuing

professional development regarding anti-racism, avoiding bias, and ensuring

equity. Principles of equitable recruitment, mentorship and advising, teaching, and

assessment should be included.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Inclusive excellence requires avoiding bias and improving racial equity; these are essential skills for faculty in

today’s teaching. Many physicians lack these skills, perpetuating health disparities, lack of diversity, and learner

mistreatment. ACGME Common Program Requirements already include specic applicable requirements.

This recommendation reinforces the importance of addressing issues related to DEI for all members of the

educational community, including residents starting from orientation. This will ultimately promote belonging,

eliminate bias, and provide social support.

UME-GME REVIEW COMMITTEE

Trustworthy Advising and

Denitive Resources

RECOMMENDATIONS

Create an interactive database with veriable GME program/track information and

make it available to all applicants, medical schools, and residency programs and at

no cost to the applicants. This will include aggregate characteristics of individuals

who previously applied to, interviewed at, were ranked by, and matched for each

GME program/track.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Veriable and trustworthy GME program/track information should be developed and made available in an easily

accessible database to all applicants. Information for the database should be directly collected and sources should

be transparent. Each program’s interviewed or ranked applicants reflect the program’s desired characteristics

more accurately than the small proportion of applicants the program matches. Data must be searchable and

allow for data analytics to assist with program decision making (e.g., allowing applicants and their advisors to input

components of their individual application to identify programs/tracks with similar current residents). Applicants

and advisors should be able to sort the information according to demographic and educational features that

may signicantly impact the likelihood of matching at a program (e.g., geography, scores, degree, visa status, etc.).

This database would also provide information on the characteristics of individuals who previously applied to and

matched into various specialties.

THEME

Evidence-informed, general career advising resources should be available for all

medical school faculty and sta career advisors, both domestic and international.

All students should have free access to a single, comprehensive electronic

professional development career planning resource, which provides universally

accessible, reliable, up-to-date, and trustworthy information and guidance. General

career advising should focus on students’ professional development; inclusive

practices such as valuing diversity, equity, and belonging; clinical and alternate

career pathways; and meeting the needs of the public. Specialty-specic match

advising should focus on the individual student obtaining an optimal match.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Centralized advising resources, developed in collaboration with specialty societies, should reflect a common core,

with supplemental information as needed, and be evidence-informed and data-driven. This will ll an information

gap and increase the transparency and reliability of information shared with students. Resources should support

the unique needs of traditionally underrepresented, disadvantaged, and marginalized student groups. Guidance

contained in the resources can support faculty in managing or eliminating conflicts of interest related to recruiting

students to the specialty, advising for the Match, and advocating for students in the Match. Advising tools

should incorporate strengths-based approaches to career selection. The resources should include the option of

nonclinical careers without stigma. Three areas of focus are envisioned: basic advising information, general career

advising, and specialty-specic match advising.

UME-GME REVIEW COMMITTEE

Trustworthy Advising and Denitive Resources

RECOMMENDATIONS

Clear and accurate information regarding clinical and nonclinical career choices should be available for all

students. The AAMC’s Careers in Medicine (CiM) platform achieves some of the aims of this recommendation.

The strengths and limitations of CiM should be examined, expanding the content and broadening access to this

resource, including to all students (U.S. MD, U.S. DO, IMG) at no cost throughout their medical school training, or at

a minimum, at key career decision-making points, in order to support students’ professional development. The

public good can be prioritized within this resource with content emphasis on workforce strategies to address

the needs of the public, including specialty selection and practice location as well as alternative nonclinical career

choices. Links to specialty-specic medical student advising resources should also be incorporated.

Basic advising information should be created for all faculty and sta who interact with students to promote

common understanding of career advising, professional development, specialty selection, and application

procedures; introduce the role of specialty-specic advisors as distinct from other faculty teachers; and minimize

sharing outdated or incorrect information with students. General career advising should be dierentiated from

specialty-specic match advising or specialty recruiting. General career advisors require expertise in career

advising; incorporate strengths-based ap-proaches to career selection including the option of nonclinical careers

without stigma; focus on professional development; value diversity, equity, and belonging; incorporate the needs

of the public; and introduce the role of specialty-specic match advisors. Specialty-specic match advisors

should undergo a training process created as part of this resource development that includes equity in advising

and mitigation of bias.

Educators should develop a salutary practice curriculum for UME career advising.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Guidelines are needed to inform U.S. MD, U.S. DO, and international medical schools in developing their career

advising programs. Standardized approaches to advising along with career advisor preparation (both general

and specialty-specic) can enhance the quality, equity, and quantity of advising and improve student trust in

the advice. Educators can improve medical student career advising by developing formal guidelines with key

recommendations based upon professional development frameworks and competencies. Implementation

of such guidelines will result in greater consistency, thoroughness, eectiveness, standardization, and equity of

medical school career advising programs to better support students in making career decisions and will lay

the foundation for career planning across the continuum.

UME-GME REVIEW COMMITTEE

Outcome Framework and

Assessment Processes

RECOMMENDATIONS

UME and GME educators, along with representatives of the full educational

continuum, should jointly dene and implement a common framework and set of

outcomes (competencies) to apply to learners across the UME-GME transition.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

A shared mental model of competence facilitates agreement on assessment strategies used to evaluate a

learner’s progress, and the inferences that can be drawn from assessments. Shared outcomes language can

convey information on learner competence with the patient/public trust in mind. For individual learners, dening

these outcomes will facilitate learning and may promote a growth mindset. For faculty, dening outcomes will

allow for the use of assessment tools aligned with performance expectations and faculty development. For

residency programs, dening outcomes will be useful for resident selection and learner handovers from UME,

resident training, and resident preparation for practice.

THEME

To eliminate systemic biases in grading, medical schools must perform initial and

annual exploratory reviews of clinical clerkship grading, including patterns of grade

distribution based on race, ethnicity, gender identity/expression, sexual identity/

orientation, religion, visa status, ability, and location (e.g., satellite or clinical site

location), and perform regular faculty development to mitigate bias. Programs

across the UME-GME continuum should explore the impact of bias on student and

resident evaluations, match results, attrition, and selection to honor societies.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Recognizing that inherent biases exist in clinical grading and assessment in the clinical learning environment, each

UME and GME program must have a continuous quality improvement process for evaluating bias in clinical grading

and assessment and the implications of these biases, including honor society selection. This recommendation is

intended to mitigate bias in clinical grading, transcript notations, MSPE reflections of remediation, and residency

evaluations. This recommendation is not intended to create requirements for reporting race, ethnicity, gender

identity, sexual identity, religion, or ability of learners as data analysis must be limited to data readily available to

each school.

The UME community, working in conjunction with partners across the continuum,

must commit to using robust assessment tools and strategies, improving upon

existing tools, developing new tools where needed, and gathering and reviewing

additional evidence of validity.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Educators from across the education continuum should use shared competency outcomes language to guide

development or use of assessment tools and strategies that can be used across schools to generate credible,

equitable, value-added competency-based information. Assessment information should be shared in residency

applications and a post-match learner handover. Licensing examinations should be used for their intended

purpose to ensure requisite competence.

UME-GME REVIEW COMMITTEE

Outcome Framework and Assessment Processes

RECOMMENDATIONS

Using the shared mental model of competency and assessment tools and

strategies, create and implement faculty development materials for incorporating

competency-based expectations into teaching and assessment.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Faculty must understand the purpose of outcomes-focused education, specic language used to dene

competence, and how to mitigate biases when assessing learners. They must understand the purpose and

use of each assessment tool. The intensity and depth of faculty development can be tailored to the amount

and type of contact that individual faculty have with students. Clerkship directors, academic progress

committees, student competency committee members, and other educational leaders require a more in-

depth understanding of the assessment system and how determinations of readiness for advancement

are made. This faculty development requires centralized electronic resources and training for trainers within

institutions. Review of training materials, and completion of any required activities to document review and/or

understanding, should be required on a regular basis.

UME-GME REVIEW COMMITTEE

Away Rotations

RECOMMENDATIONS

Convene a workgroup to explore the multiple functions and value of away rotations

for applicants, medical schools, and residency programs. Specically, consider

the goals and utility of the experience, the impact of these rotations, and issues of

equity including accessibility, assessment, and opportunity for students from groups

underrepresented in medicine and nancially disadvantaged students.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Away rotations can be cost prohibitive yet may allow a student to get to know a program, its health system, and

surrounding community. Some programs are reliant on away rotations to showcase their unique strengths to

attract candidates. Given the multifactorial and complex role that away rotations fulll, a committee should be

convened to conduct a thorough and comprehensive review of cost versus benet of away rotations, followed

by recommendations from that review. Non-traditional methods of conducting and administering away

rotations should be explored (e.g., oering virtual away rotations, waiving application fees, or oering away

stipends particularly for nancially disadvantaged students).

THEME

UME-GME REVIEW COMMITTEE

Equitable, Mission-Driven

Application Review

RECOMMENDATIONS

A convened group including UME and GME educators should reconsider the

content and structure of the MSPE as new information becomes available to

improve access to longitudinal assessment data about applicants. Short-term

improvements should include structured data entry elds with functionality to

enable searching.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

The development of UME competency outcomes to apply across learners and the continuum is essential in

decreasing the reliance on board scores in the evaluation of the residency applicant. These will take time to

develop and implement and may be developed at dierent intervals. As new information becomes available

to improve applicant data, the MSPE should be utilized to improve longitudinal applicant information. In addition,

improvements in the MSPE, such as structured data entry elds with functionality to enable searching, should be

explored.

THEME

Structured Evaluative Letters (SELs) should replace all Letters of Recommendation

(LORs) as a universal tool in the residency program application process.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

A Structured Evaluative Letter (SEL), which would include specialty-specic questions, would provide knowledge

from the evaluator on student performance that was directly observed versus a narrative recommendation.

The template should be based on an agreed upon set of core competencies and allow equitable access to

completion for all candidates. The SEL should be based on direct observation and must focus on content that

the evaluator can complete. Faculty resources should be developed to improve the quality of the standardized

evaluation template and decrease bias.

To raise awareness and facilitate adjustments that will promote equity and

accountability, self-reported demographic information of applicants (e.g., race,

ethnicity, gender identity/expression, sexual identity/orientation, religion, visa

status, or ability) should be measured and shared with key stakeholders, including

programs and medical schools, in real time throughout the UME-GME transition.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Inequitable distribution of applicants among specialties is not in the best interest of programs, applicants, or the

public good. Bias can be present at any level of the UME-GME transition. A decrease in diversity at any point along

the continuum provides an important opportunity to intervene and potentially serve the community in ways

that are more productive. An example of accountability and transparency in an inclusive environment across the

continuum is a diversity dashboard for residency applicants. A residency program that nds bias in its selection

process could go back in real time to nd qualied applicants who may have been missed, potentially improving

outcomes.

UME-GME REVIEW COMMITTEE

Equitable, Mission-Driven Application Review

RECOMMENDATIONS

To optimize utility, discrete elds should be available in the existing electronic

application system for both narrative and ordinal information currently presented

in the MSPE, personal statement, transcript, and letters. Fully using technology

will reduce redundancy, improve comprehensibility, and highlight the unique

characteristics of each applicant.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Optimally, each applicant will be reviewed individually and holistically to evaluate merit. However, some

circumstances may require rapid review. The 2020 NRMP program directors’ survey found that only 49% of

applications received an in-depth review. The application system should utilize modern technology to maximize

the likelihood that applications are evaluated in a way that is holistic, mission-based, and equitable.

Currently, applications are assessed based on the information that is readily available, which may place undue

emphasis on scores, geography, medical school, or other factors that perpetuate bias. Adding specic data

gives an opportunity for applicants to demonstrate their strengths in a way that is user-friendly for program

directors. Maximizing the amount of accurate information readily available in the application will increase

capacity for holistic review of more applicants and improve trust during the UME to GME transition. Although

not all schools and programs will align on which information should be included, areas of agreement should be

identied and emphasized.

To promote equitable treatment of applicants regardless of licensure examination

requirements, comparable exams with dierent scales (COMLEX-USA and USMLE)

should be reported within the electronic application system in a single eld.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Osteopathic medical students make up 25% of medical students in U.S. schools and these students are required

to complete the COMLEX-USA examination series for licensure. Residency programs may lter out applicants

based on their USMLE score leading many osteopathic medical students to sit for the USMLE series. This creates

substantial increase in cost, time, and stress for osteopathic students who believe duplicate testing is necessary

to be competitive in the Match. A combined eld should be created in the Electronic Residency Application

Service (ERAS) that normalizes the scores between the two exams and allows programs to lter based only on

the single normalized score. This will mitigate structural bias and reduce nancial and other stress for applicants.

Filter options available to programs for sorting applicants within the electronic

application system should be carefully created and thoughtfully reviewed to ensure

each one detects meaningful dierences among applicants and promotes review

based on mission alignment and likelihood of success at a program.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Currently, residency programs receive more applications than they can meaningfully review. For this reason,

lters are sometimes used to identify candidates that meet selection criteria. However, some commonly used

lters may exclude applicants who are not meaningfully dierent from ones who are included (e.g., students

who took a dierent licensure examination, students with statistically insignicant dierences in scores, students

from dierent campuses of the same institution, etc.). The use of free text lters increases the risk of not

identifying, or mischaracterizing applicant characteristics. All applications should be evaluated fairly, independent

of software idiosyncrasies. Filters should be developed in conjunction with all stakeholders. Each lter that is

oered should align with the missions and requirements of residency programs.

UME-GME REVIEW COMMITTEE

Equitable, Mission-Driven Application Review

RECOMMENDATIONS

Convene a workgroup of educators across the continuum to begin planning for a

dashboard/portfolio to collect assessment data in a standard format for use during

medical school and in the residency application process. This will enable consistent

and equitable information presentation during the residency application process

and in a learner handover.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Key features of a dashboard/portfolio in the UME-GME transition, and across the continuum, should include

competency-based information that aligns with a shared mental model of outcomes, clarity about how

and when assessment data were collected, and narrative data that uses behavior-based and competency-

focused language. Learner reflections and learning goals should be included. Dashboard development will

require careful attention to equity and minimizing harmful bias, as well as a focus on the competencies

and measurements that predict future performance with patients. Transparency with students about the

purpose, use, and reporting of assessments, as well as attention to data access and security, will be essential.

UME-GME REVIEW COMMITTEE

Optimization of Application,

Interview, and Selection Processes

RECOMMENDATIONS

All interviewing should be virtual for the 2021-2022 residency selection season. To

ensure equity and fairness, there should be ongoing study of the impact of virtual

interviewing as a permanent means of interviewing for residency.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Virtual interviewing has had a signicant positive impact on applicant expenses. With elimination of travel, students

have been able to dedicate more time to their clinical education. Due to the risk of inequity with hybrid interviewing

(virtual and in person interviews occurring in the same year or same program), all interviews should be conducted

virtually for the 2021-2022 season. Hybrid interviewing (virtual combined with onsite interviewing) should be

prohibited.

A thorough review of the data around virtual interviewing is also recommended. Candidate accessibility, equity,

match rates, and attrition rates should be evaluated. Residency program feedback from multiple types of

residencies should be solicited. In addition, the separation of applicant and program rank order list deadlines in time

should be explored, as this would allow students to visit programs without pressure and minimize influence on a

program’s rank list.

THEME

Develop and implement standards for the interview oer and acceptance

process, including timing and methods of communication, for both learners and

programs, to improve equity and fairness, to minimize educational disruption, and

to improve wellbeing.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

The current process of extending interview oers and scheduling interviews is unnecessarily complex and onerous,

with little to no regulation. Applicant stress and loss of rotation education while attempting to conform to some

elements (e.g., obsessively checking emails to accept short-timed interview oers) can be improved with changes

to the application platform, policies, and procedures. Development of a common interview oering/scheduling

platform and creating policies (e.g., forbidding residency programs to over oer/over schedule interviews and

from setting inappropriate time-based applicant replies), would result in important improvements. While these

processes are being developed, residency programs involved in the 2021-2022 residency selection season should

allow applicants 24 to 48 hours to accept or decline an interview oer. In addition, for the 2021-2022 residency

selection season, programs should not oer more interviews to applicants than available interview positions.

Likewise, applicants should not accept multiple interviews that are scheduled at the same time.

UME-GME REVIEW COMMITTEE

Optimization of Application, Interview, and

Selection Processes

RECOMMENDATIONS

Innovations to the residency application process should be piloted to reduce

application numbers and concentrate applicants at programs where mutual

interest is high, while maximizing applicant placement into residency positions. Well-

designed pilots should receive all available support from the medical community

and be implemented as soon as the 2022-2023 application cycle; successful pilots

should be expanded expeditiously toward a unied process.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Application inflation is a major problem in the current dysfunction in the UME-GME transition. The 2020 NRMP

program director’s survey found that only 49% of applications received an in-depth review; an unread

application represents wasted time and expense for applicants. Yet doubling the program resources available

for review is not practical. Informational interventions – like improved career advising and transparency – are

unlikely to reduce application numbers signicantly in the context of a high stakes prisoner’s dilemma. In sum,

the current process is costly to applicants and program directors and does not optimally serve the public good.

To address this dysfunction, Coalition organizations and other groups in the medical community should utilize

all available logistic, analytic, and nancial resources to lead and support innovative pilots to reduce application

numbers and concentrate applicants at programs where mutual interest is high, while maximizing applicant

placement into residency positions. Pilots should be based on best available evidence, specialty-specic needs,

potential impact (both positive and negative), and collaboration among stakeholders. Pilot innovations, some

of which are ongoing, could include, but are not limited to, the following: expanding integrated UME-GME

pathways, preference signaling, application caps, and/or additional application or match rounds.

Groups sponsoring pilots should be accountable for using a continuous quality improvement approach to

gather and monitor evidence of eectiveness and equity across applicant groups with historically distinct

application behaviors and outcomes, including United States MD and DO graduates, international medical

graduates, couples applicants, previously unmatched applicants, and individuals belonging to groups that are

underrepresented in medicine.

While pilot studies may vary across specialties, ultimately the redesigned residency application process should

be as consistent as possible across specialties, recognizing that applicants, advisors, and program directors

may be subject to the rules of multiple specialties in the context of combined tracks, couples, and dual

applicants.

Implement a centralized process to facilitate evidence-based, specialty-specic

limits on the number of interviews each applicant may attend.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Identify evidence-based, specialty-specic interview caps, envisioned as the number of interviews an applicant

attends within a specialty above which further interviews are not associated with signicantly increased match

rates, across all core applicant types. Create a centralized process to operationalize interview caps, which could

include an interview ticket system or a single scheduling platform.

UME-GME REVIEW COMMITTEE

Educational Continuity and

Resident Readiness

RECOMMENDATIONS

Early and ongoing specialty-specic resident assessment data should be

automatically fed back to medical schools through a standardized process to

enhance accountability and to inform continuous improvement of UME programs

and learner handovers.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Instruments for feedback from GME to UME should be standardized and utilized to inform gaps in curriculum and

program improvement. UME institutions should respond to the GME feedback on their graduates’ performance in

a manner that leads to quality improvement of the program.

THEME

Develop a portfolio of evidence-based resident support resources for program

directors, designated institutional ocials (DIOs), and residency programs. These will

be identied as salutary practices, and accessible through a centralized repository.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

A centralized source of resident support resources will assist programs with eective approaches to address

resident concerns. This will be especially relevant for competency-based remediation and resident wellbeing

resources in the context of increased demand for support around the UME-GME transition. Access for programs

and program directors will be low/no cost, condential, and straightforward.

Targeted coaching by qualied educators should begin in UME and continue during

GME, focused on professional identity formation and moving from a performance

to a growth mindset for eective lifelong learning as a physician. Educators should

be astute to the needs of the learner and be equipped to provide assistance to all

backgrounds.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Coaching can benet a student’s transition to become a master adaptive learner with a growth mindset. While

this transition should begin early in medical school, it should be complete by the time that the student moves from

UME to GME. If a learner does not transition to a growth mindset, their wellness and success will be compromised.

The addition of specic validated mentoring programs (e.g., Culturally Aware Mentoring) and formation of anity

groups to improve sense of belonging should be considered.

UME-GME REVIEW COMMITTEE

Educational Continuity and Resident Readiness

RECOMMENDATIONS

Specialty-specic, just-in-time training must be provided to all incoming rst-year

residents, to support the transition from the role of student to a physician ready to

assume increased responsibility for patient care.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

The intent of this recommendation is to level set incoming resident preparation regardless of medical school

experience. Recent research has shown that residents reported greater preparedness for residency if they

participated in a medical school “boot camp,” and participation in longer residency preparedness courses

was associated with high perceived preparedness for residency. This training must incorporate all six specialty

competency domains and be conducive to performing a baseline skills assessment. These curricula might

be developed by specialty boards, specialty societies, or other organized bodies. To minimize costs, specialty

societies could provide centralized recommendations and training could be executed regionally or through

online modules.

Residents must be provided with robust orientation and ramp up into their

specic program at the start of internship. In addition to clinical skills and system

utilization, content should include introduction to the patient population, known

health disparities, community service and engagement, faculty, peers, and

institutional culture.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Improved orientation to residency has the potential to enhance trainee wellbeing and improve patient safety.

Residents should have orientation that includes not only employee policies, but also education that optimizes

their success in their specic clinical environment. Residents, like other employees, should be paid for attending

orientation.

Meaningful assessment data based on performance after the MSPE must

be collected and collated for each graduate, reflected on by the learner with

an educator or coach, and utilized in the development of a specialty-specic,

individualized learning plan to be presented to the residency program to serve as a

baseline at the start of residency training.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Guided self-assessment by the learner is an important component in this process and may be all that is

available for some international medical graduates. This recommendation provides meaning and importance

for the assessment of experiences during the nal year of medical school (and possibly practice for some

international graduates), helps to develop the habits necessary for life-long learning, and holds students

and schools accountable for quality senior experiences. It also uses the resources of UME to prepare an

individualized learning plan (ILP) to serve as a baseline at the start of GME. This initial ILP will be rened by

additional assessments envisioned as an “In-Training Examination” (ITE) experience early in GME. The time for

this experience should be protected in orientation, and the feedback should be formative similar to how most

programs manage the results of ITEs. This assessment might occur in the authentic workplace and based

on direct observation or might be accomplished as an Objective Structured Clinical Exam using simulation.

This assessment should inform the learner’s ILP and set the stage for the work of the clinical competency

committee of the program.

UME-GME REVIEW COMMITTEE

Health and Wellness

RECOMMENDATIONS

Anticipating the challenges of the UME-GME transition, schools and programs

should ensure that time is protected, and systems are in place, to guarantee that

individualized wellness resources – including health care, psychosocial supports, and

communities of belonging – are available for each learner.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

Given that the wellness of each learner signicantly impacts learner performance, it is in the program and public’s

best interest to ensure the learner is optimally prepared to perform as a resident. There should be a focus on

applying resources that are already available rather than depending on the creation of new resources. Examples

of wellness resources include enrollment in health insurance, establishing with a primary care provider and dentist,

securing a therapist if appropriate, identifying local communities of belonging, and other supports that optimize

wellbeing. These resources may especially benet the most vulnerable trainees.

THEME

Adequate and appropriate time must be assured between graduation and learner

start of residency to facilitate this major life transition.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

The transition from medical school to residency typically marks a concrete transition from paying for education

to becoming a fulltime employee focused on the lifelong pursuit of professional improvement. This transition is life

changing for many. It often requires a move from one location to another, sometimes across the world. There must

be time for licensing and in some cases, visa attainment. Often this life transition is accompanied by other major

life events such as partnering or childbearing. Once residency starts, the learner may work many hours each

week and may have little time to establish a home. Thus, it is important for wellness and readiness to practice that

adequate time be provided to accomplish this major life transition.

The predictability of this transition must be recognized by both UME and GME institutions, and cooperation on both

sides is required for this transition to be accomplished smoothly. There is a desire to overall better prepare learners

for the start of residency, and an assured transition time would allow related recommendations to be more easily

accomplished.

All learners need equitable access to adequate funding and resources for the

transition to residency prior to residency launch.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

As almost every learner graduating from medical school transitions to residency, the need to fund a geographic

move and establishment of a new home is predictable. This nancial planning should be incorporated into medical

school expenses, for example through equitable low interest student loans. Options to support the transitional

expenses of international medical graduates should also be identied. These costs should not be incurred by GME

programs.

UME-GME REVIEW COMMITTEE

Health and Wellness

RECOMMENDATIONS

There should be a standardized process throughout the United States for initial

licensing at entrance to residency to streamline the process of credentialing for both

residency training and continuing practice.

NARRATIVE DESCRIPTION OF RECOMMENDATION:

To benet the public good, costs to support the U.S. healthcare workforce should be minimized. To this end, all

medical students should be able to begin licensure earlier in their educational continuum to better distribute

the work burden and costs associated with this predictable process. When learners are applying to programs

in many dierent states, the varied requirements are unnecessarily cumbersome. Especially for states where

a training license is required, the time between the Match and the start of the rst year of residency is often

inadequate for this purpose. This is a potential cost saving measure.

UME-GME REVIEW COMMITTEE

Throughout the transition, learners are being

assessed, including through preclinical course

examinations, rotation evaluations, and

licensure examinations

Medical School

Graduation

20 19 17

Oered and

Partake in

Interviews

24 22

Virtual Interviews for

2021-2022

Standards for

Interview Oer and

Acceptance Process

Interview Limits

Standardized

Process for Initial

Licensing

Residency

Preparedness

28 29 30

Specialty Specic

Residency

Preparation

Improved Residency

Program Orientation

UME to GME ILP

Hando

Initiate

Support

Structures

Centralized Resident

Support Resources

Wellness Resources

for the Transition

Major Life

Transitions

Apply to Many

Programs

Residency Application

Process Innovations

Assured Time Between

UME and GME

Equitable Access to

Funding for Transition

Embed in

Learning and

Work

Environment

Professional

Identity

Formation

The UME-GME transition encompasses a complex ecosystem. As

learners navigate through the UME-GME transition, they interact with

numerous organizations with jurisdiction over specic components of

the process. However, the ecosystem is not governed by a single entity.

9 11 12

Competency Based

Faculty Development

Materials

Improved

Assessment Tools

Common

Competencies

across Transition

Feedback from

GME to UME

Committee to

Manage CQI across

Transition

Residency Selection

and Physician

Workforce Research

IRP Reform

The

Learner’s

Journey

Collaboration and Continuous

Quality Improvement

Diversity, Equity and Inclusion

Trustworthy Advising and

Denitive Resources

Outcome Framework and

Assessment Processes

Away Rotations

Equitable Mission-Driven

Application Review

Optimizing Application, Interview

and Selection Processes

Educational Community and

Resident Readiness

Health and Wellness

Medical schools and

programs have a

responsibility to promote

equity and diversity across

the continuum

10 5 4

Specialty Specic

Practices to

Increase Diversity

DEI Education

Across the

Continuum

CQI to Mitigate Bias

across Transition

Sharing Applicant

Demographics to

Improve Diversity

Career

Advising

Resources

Career

Advising

Curriculum

Coaching for

Professional

Identity

Formation

Prepare Application

Materials

Research

Residency

Programs

Final

Year

Review of Away

Rotations

18 15 14

Interactive

GME

Database

Electronic Application

System Improvements

Review of Filter

Content and Use

Standardized Dashboard

and Portfolio for Learners

MSPE Revision

Structured

Evaluative Letters

Reporting Licensure Exams

in a Single Field

Selection

Process

Hiring and

Credentialing

UGRC Process

In the summer of 2020, a Planning Committee of the Coalition selected the

members of the UGRC and charged them with the task of recommending solutions

to identied challenges in the transition. The following is a description of the work

process of the UGRC.

ORIGIN OF THE UGRC

In 2018, a national conversation culminated regarding the use of numeric scores associated with medical

licensing examinations in residency applicant screening and selection. In response, the chief executive ocers of

ve national organizations agreed to co-sponsor an Invitational Conference on USMLE Scoring (InCUS) in March

2019, with the primary goal of reviewing the practice of numeric score reporting. Three recommendations that

emerged focused on the USMLE, however the fourth InCUS recommendation focused on the UME-GME transition:

Convene a cross-organizational panel to create solutions for the assessment and transition challenges from UME

to GME.

In September 2019, a proposal was made to the Coalition to convene a UME-GME Review Committee in line with

the fourth recommendation from InCUS. The Coalition’s members are the national organizations responsible for

the oversight, education and assessment of medical students and physicians throughout their medical careers.

As a result, a Planning Committee was created by the Coalition to develop the construct, membership, and charge

of the Review Committee, which would be responsible for recommending solutions to identied challenges in

the UME-GME transition.

In January 2020, a call for nominations was issued for individual representatives to the

Planning Committee from undergraduate medical educators, residency program directors, learners, and the

public. The Coalition’s Management Committee selected the individual members of the Planning Committee from

over 60 responses. In addition, organizational representatives from AACOM, AAMC, AOGME, ECFMG, NBME, NBOME,

and OPDA were appointed to the Planning Committee.

The Planning Committee met in March 2020 and identied the construct and structure of the UGRC, developed

a process for selecting its members, and determined the key questions that the UGRC should consider. The

Planning Committee discussed the scope of the UGRC and organized pertinent issues into three broad themes:

(a) preparation and selection for residency, (b) the application process, and (c) overall considerations such as

diversity and specialty specic competencies. The Planning Committee also spelled out the timeline, deliverables,

expectations, and composition of the UGRC. An open call for nominations took place in May and June of 2020 and

the Planning Committee reviewed 183 applications to populate a balanced UGRC that included undergraduate

and graduate medical educators, organizational members, public members, students, and residents. Care was

taken to ensure that multiple perspectives would be represented on the UGRC, including type of degree (DO

and MD), racial and ethnic diversity, range of specialties, geographic distribution, and persons with a focus on

undergraduate medical education (faculty and deans) and graduate medical education (program directors and

DIOs). All UGRC members were selected in July, the co-chairs were named in August, and the UGRC held its rst

meeting in September 2020.

COALITION FOR PHYSICIAN ACCOUNTABILITY

UME-GME REVIEW COMMITTEE

UGRC STRUCTURE

The UGRC was led by an Executive Committee comprised of the two co-chairs, the lead Coalition sta member,

and the four original workgroup leads. The co-chairs and lead sta member initially created four workgroups to

optimize group dynamics and distribute Committee work in an organized fashion. Because the charge from the

Planning Committee included an ambitious start-to-nish timeline (September 2020 to June 2021), this structure

allowed groups to work in parallel and delve more deeply into assigned tasks. Beyond the individual workgroup

areas of focus, all workgroups also included four overall cross-cutting themes throughout their deliberations:

diversity, equity, inclusion, and fairness; wellbeing; specialty focus; and the public good. Finally, the four original

workgroups were asked to develop research questions and to consider next steps after the UGRC completed its

charge, both of which were needed to help implement nal recommendations and to inform future discussions

out of scope of the UGRC. In February 2021, the co-chairs created a fth workgroup to ensure that the UGRC

appropriately addressed the critical issues of diversity, equity, and inclusion (DEI). Finally, in April 2021, the co-

chairs tasked a sixth “bundling workgroup” to consolidate similar recommendations, sequence interdependent

recommendations, and re-organize the nal recommendations into more cogent themes.

The UGRC was assisted in its work process by generous sta support from Coalition member organizations,

including a project manager, communications director, medical writer, survey analysts, and graphics designers.

Medical librarians searched the literature to support an evidence-informed approach.

UGRC Workgroup Focus Areas

Workgroup A: Ensuring Residency Readiness

General competencies

Selection of residency/specialty eld

Workgroup B: Mechanics of the Application/Selection Process from the UME Perspective

Information sharing

Application content

Application mechanics

Workgroup C: Mechanics of the Application/Selection Process from the GME Perspective

Information sharing

Application process

Interviewing

The Match

Workgroup D: Post-Match Optimization

Optimizing UME by enhancing residency readiness

Optimizing GME by ensuring patient safety

Information sharing

Feedback to UME

DEI Workgroup: Focus on Diversity, Equity, and Inclusion

Bundling Workgroup: Consolidation, Sequencing, and Reorganization of Themes

UME-GME REVIEW COMMITTEE

FOUNDATIONAL WORK PROCESS OF THE UGRC

Between September 2020 and June 2021, the entire UGRC met virtually on six separate occasions. Each of

these meetings consisted of multiple sessions spread over two or three days. In addition, a special session of the

UGRC occurred on April 5, 2021, to reconsider several initial recommendations. In between the full Committee

meetings, each workgroup met intermittently to fulll its tasks. A summary was widely distributed to the public

after each Committee meeting to update the community on the UGRC’s progress to date. Further, the UGRC

issued three explicit calls for external stakeholder engagement. The rst one occurred in December 2020 and

focused on envisioning the ideal state of the UME-GME transition. The second occurred in March 2021 and focused

on descriptions of current innovations to improve the UME-GME transition. The third opened in late April 2021 and

specically asked for feedback on the UGRC preliminary recommendations.

The rst virtual meeting of the UGRC occurred in September 2020. Seven consensus ideas quickly emerged

on how to manage the work of the Committee. First, the members agreed that the UME-GME transition

encompassed far more than preparation, application, and selection for residency. This led to an elaboration of

the charge to include both optimal preparation for caring for patients early in residency as well as considerations

on how to leverage learners’ time and experiences between the Match and the initial months of training. In other

words, the successful transition requires adopting and valuing a growth mindset, accompanied by a dramatic

change in focus where the emphasis shifts away from being student-centric and towards being patient-centric.

Second, it was evident that level setting was needed to ensure that all UGRC members had common

understanding of the UME-GME transition because not all UGRC members were knowledgeable about each

aspect and component of the transition ecosystem. To address this problem, the co-chairs called upon members

of the UGRC, or in some cases employees of Coalition organizations, to create a series of video presentations (i.e.,

voice over power points) that members could watch asynchronously. The video presentations helped all the UGRC

members reach a baseline level of understanding about the transition.

Third, there was a strong sentiment that the Committee should approach its work using the concept of backward

design (i.e., rst imagine an idealized desired state and then think about how to create a system that produces

it). Each UGRC workgroup spent two months envisioning an idealized state for their area of focus, and then the

workgroup leaders harmonized them into a single ideal state for the UME-GME transition. As described earlier in

this report, the nished product included elements of the overall ecosystem and addressed wellness, specialty

selection, learner selection, competence, continuum and hando, technology, licensing and credentialing, life

transition, residency launch, and residency environment. This exercise allowed the UGRC to articulate a blue-sky

denition of success for an equitable, ecient, and transparent system across the UME-GME transition.

In December 2020, the UGRC released a survey designed to engage external stakeholder organizations about

what should be included in the ideal state. Thirty-two organizations responded to the survey and the ideas they

shared were organized into eight themes, each of which had been identied by the UGRC workgroup leads when

creating the Committee’s harmonized ideal state. Thus, this rst call for stakeholder engagement did not result in

any substantive changes to what the UGRC had created. The UGRC’s shared vision of the ideal state has guided its

ongoing work.

The fourth consensus idea was that the UGRC should approach the identied challenges in a systematic manner

to unearth the root causes of problems with the current UME-GME transition. Thus, four workgroups spent many

weeks discussing why identied challenges existed and why they persisted. This series of exercises produced

workgroup-specic Ishikawa diagrams (i.e., shbones) that identied myriad problems underlying the identied

challenges associated with the transition. Each shbone was presented to the entire Committee so that UGRC

members could reflect on the problems found by each of the four workgroups. To ensure that the root cause

analyses were sound, UGRC members responded to a series of provocative questions designed to challenge

common assumptions about the transition, and they were then asked to rate which problems were most

important to address. The Ishikawa diagrams are included in Appendix B of this report.

Importantly, the fth consensus idea that the UGRC agreed upon was to avoid premature discussion or advocacy

for any specic solution to the identied challenges of the UME-GME transition. The idea was simple: articulate

the desired outcome and understand the root problems before generating solutions. Discussion about possible

remedies was not permitted until the UGRC had created a shared ideal state for the UME-GME transition and each

workgroup had completed its root cause analysis (i.e., Ishikawa diagram). Indeed, even after both exercises were

nished, the UGRC took the time to examine the ecosystem for components of the current UME-GME transition

that worked well. This exercise helped identify current aspects and processes that should be preserved.

In January 2021, the UGRC began to brainstorm solutions to the root causes identied by the workgroups. These

brainstorming sessions occurred in both the workgroups and meetings of the entire Committee. The UGRC used

a virtual white board to help with discussion, dissection, debate, and renement of ideas before they could be

incorporated into recommendations. At this stage, the UGRC’s sixth consensus idea was set into motion, which

was simply “to not reinvent the wheel.” Thus, a concerted eort was made to identify potential solutions and

innovations described in the literature or implemented by institutions across the country.

In February 2021, the UGRC released a second call for external stakeholder input. This eort to engage

stakeholders asked individuals and organizations to share innovations that had been implemented to address

concerns about the UME-GME transition. In total, 35 responses containing 39 self-described innovations were

submitted for review to the UGRC. Of note, a majority of the innovations submitted through this process had

previously been identied by the Committee.

Lastly, the seventh consensus idea was to strive to be evidence-based whenever possible. To that end, the

UGRC secured the services of three research librarians who could search the literature and public databases

when a member or a workgroup had a question about an issue. UGRC members had hopes of generating

recommendations that were data-driven and evidence-based. However, relatively few aspects of the UME-GME

transition have undergone systematic review. Similarly, many innovations described in the literature are descriptive

in nature without generalizable outcomes. This led the co-chairs to embrace a consensus approach to endorsing

recommendations, informed by available evidence, as opposed to identifying evidence-based recommendations.

GENERATION AND ADOPTION OF RECOMMENDATIONS

By February 2021, the workgroups had begun the process of forming preliminary recommendations for the entire

UGRC to consider. As those eorts progressed, the workgroup leaders identied two issues that required attention

by the co-chairs. The rst was to provide a forum for contentious issues to be discussed by the full UGRC, and the

second was to provide guidance regarding the level of granularity for the recommendations. To address the rst

concern, the co-chairs asked each workgroup leader to select a few recommendations that might generate

disagreement, and the majority of the February UGRC meeting was devoted to discussion and debate about

these topics. To address the second issue, a template was created that included instructions on how to frame

each recommendation in broad terms, and to include specic examples on how a recommendation might be

implemented.

The initial recommendation template was designed to be comprehensive and included the following ten elds:

recommendation; narrative description; specic examples of how the recommendation might be implemented;

questions for librarians; known citations or references; organizations or stakeholders that could help implement

the recommendation; links to the ideal state and Ishikawa diagrams; cross-cutting themes that are impacted;

potential desired outcomes and consequences; potential barriers to implementation; and future research

questions. The co-chairs later created a streamlined version of the templates that accompany each of the

UGRC’s nal recommendations. All 34 templates are included in Appendix C. Importantly, the templates provide

essential background information, supporting evidence, important context, and the rationale for each UGRC

recommendation.

COALITION FOR PHYSICIAN ACCOUNTABILITY

As the groups worked to rene their recommendations and complete the templates, the co-chairs devised a

process for sharing, presenting, adopting, reconsidering, and editing the preliminary recommendations from each

workgroup. The co-chairs determined that a super majority of 67% (two-thirds of the members present) would

be required to adopt a recommendation, and that the process would allow any member who had concerns

to bring them forward and propose edits that would facilitate a vote to adopt. In other words, the underlying

philosophy was for the Committee to “get to yes” and achieve a high degree of consensus. Importantly, each

recommendation brought to the full Committee was sponsored by one of the workgroups, whose members had

more thoroughly debated and thought through pertinent issues.

The UGRC met virtually in March 2021, to take decisional votes on each recommendation proposed by the four

main workgroups. In total, the workgroup leaders presented 41 recommendations to the UGRC. Each presentation

included (a) the recommendation, (b) the narrative description, (c) components that each recommendation

required (i.e., “must haves”) as well as those that would be “nice to have,” and (d) a table outlining pros and cons

of the recommendation. The presentation was followed by a facilitated discussion that allowed members to

ask questions, seek clarications and raise concerns about the proposed recommendation. Potential edits

to the recommendation were also entertained, followed by a binding vote to either adopt or not adopt the

recommendation as written. Of the 41 recommendations initially presented, 36 were adopted with at least a 67%

majority, and ve were not adopted.

Workgroups that had proposed a recommendation that was not adopted were given the option of altering the

recommendation and asking for the modied recommendation to be reconsidered. In addition, every member

was allowed to propose new recommendations. However, only the DEI workgroup used that mechanism to

propose new recommendations. The new recommendations, together with the recommendations being

reconsidered, were processed in the same manner as the original 41 recommendations (i.e., a preliminary vote,

presentation of the recommendation, facilitated discussion, and entertainment of suggested edits). When the

UGRC convened for a special session in April 2021, six more recommendations were adopted (three altered

recommendations brought back for reconsideration and three new recommendations related to diversity, equity,

and inclusion).

In total, the UGRC adopted 42 preliminary recommendations, organized under 12 themes: oversight; advising of

learners; competencies and assessments; away rotations; diversity, equity, and inclusion in medicine; application

process; interviewing; matching process; faculty support resources; post-match transition to residency; policy

implications; and research questions. The preliminary recommendations and pertinent background material were

presented to the Coalition in April 2021, followed one week later by their widespread release and a one-month call

for public comment. The preliminary recommendations of the UGRC can be found in Appendix D.

The solicitation for public comment was facilitated by the creation of a digital survey instrument with a prominent

link on the Coalition’s website. The link was made widely available to interested parties and all stakeholder groups.

The call for public comment was disseminated through numerous communication channels including social

media platforms, email distribution lists, outreach presentations, and individual networks. In addition, periodic

reminders were issued throughout the open call to increase the number of responses

In total, the survey instrument collected 2,673 comments from 768 distinct respondents during the period of time

that the survey was administered. Of these responses, 13.7% were submitted on behalf of an organization or

group in an ocial capacity, which accounted for 21.2% of the overall comments. The survey responses were

analyzed as follows by a team from the NBME with expertise in qualitative and quantitative methods.

Prior to the survey administration window, UGRC stakeholders were asked to provide a list of potential codes

or topics that would likely be discussed in the respondents’ comments. After the rst week of the survey

administration window, four NBME sta members read portions of the response data and identied a list of

potential thematic codes. The list of codes was presented to UGRC stakeholders for review and approval. The

four NBME sta members then coded the rst two weeks of comments using the initial codebook. Subsequently,

through an iterative process, additional codes and tags were added, which resulted in a nal set of agreed-upon

codes and tags. The nal codebook was used by the NBME sta members to code the remainder of responses

in weekly batches. Two NBME sta members reviewed 10% of all coded comments from the rst two weeks of

COALITION FOR PHYSICIAN ACCOUNTABILITY

UME-GME REVIEW COMMITTEE

the survey window to ensure that codes were being adequately and accurately used. This review resulted in the

application of additional codes to the comments and not to the deletion of previously applied codes. Through

discussion, NBME sta members also attended to their reactions to the responses, their backgrounds, and their

potential biases. To clarify relationships between associated codes, codes were organized using a parent-child

code structure in which a parent code could include any number of subcategories, or “children.” In all tables and

gures in the results section, an asterisk was used to indicate which of the codes are parent codes. If a child

code was applied to a free-text response, its parent code was also applied or “upcoded.” All free-text responses

were also assigned sentiment (agree, disagree, or mixed) when distinct sentiment was expressed in a comment.

Additionally, a list of tags was applied to all free-text responses when applicable. The full report from the team was

made available to all Committee members before the UGRC’s recommendations were nalized. This report from

the NBME team can be found in Appendix E.

To prepare for the June 2021 UGRC meeting, multiple members of the UGRC’s Executive Committee read the

survey report in full, and ve workgroup leaders were assigned to summarize commentary about each of the

preliminary recommendations. In addition to the information contained in the survey report, feedback from the

organizational members of the Coalition and input obtained by the co-chairs through dialogue with students,

program directors, DIOs, medical educators, medical school deans, and international medical graduates was

shared with each member of the UGRC to inform the Committee’s nal recommendations. During the June

meeting all stakeholder feedback, strategies for consolidating and sequencing the recommendations, and

reconsidered themes were presented, discussed, and nalized. New language for the recommendation

addressing application inflation was also proposed, discussed, and adopted. As a result of these eorts, the UGRC

adopted 34 nal recommendations organized around nine themes. Moreover, the recommendations within each

theme are sequenced in chronologic order to guide their implementation.

FINAL STEPS

The Executive Committee was responsible for writing this nal report on behalf of the UGRC. The report includes

the templates created by the workgroups as well as input from all members of the Committee. There are ongoing

discussions regarding possible opportunities for scholarly activity with the purpose of codifying and further

sharing the work of the Committee. The UGRC co-chairs will deliver this nal report to the Coalition in mid-July

2021, and the UGRC will disband shortly thereafter. The Coalition will then meet in late July to consider adoption of

the recommendations and determine next steps towards implementation.

UME-GME REVIEW COMMITTEE

Future Ideal State

From the outset, the UGRC agreed to envision an idealized future state for the

transition from UME to GME before developing any recommendations. The idea

was to use the concept of backward design so that the UGRC could identify the

characteristics of a system that would create that ideal state. What would success

look like if the transition worked as a cohesive ecosystem that served all learners,

faculty, clinical supervisors, and patients?

Beginning with the rst virtual meeting of the UGRC in September 2020, four UGRC workgroups spent two months

conceptualizing an idealized state for their area of focus:

• Ensuring Residency Readiness

• Mechanics of the Application/Selection Process from the UME Perspective

• Mechanics of the Application/Selection Process from the GME Perspective

• Post-Match Optimization

The workgroup leaders then harmonized each component into a composite ideal state for the UME-GME

transition. Soon thereafter, a public comment period was opened to solicit additional ideas from external

stakeholders. In total, 32 organizations responded. Overall, the stakeholder input armed the concepts developed

by the UGRC and led to an improvement in clarity and wording but did not result in substantive content changes to

the UGRC’s proposed ideal state. The nalized composite ideal state for the UME-GME transition guided the UGRC’s

ongoing work and addressed the following areas: wellness, specialty selection, learner selection, competence,

continuum, hando, technology, licensing, credentialing, life transition, residency launch, and residency environment.

THE IDEAL STATE

Overall

The foundation of the ideal state as envisioned by the UGRC is a set of core values and concepts. The ideal UME-

GME transition is equitable, coordinated, ecient, transparent, and cohesive. It is an ecosystem that supports

each learner’s growth, evidence-informed specialty selection, achievement of competence, and maintenance

and improvement of wellness. Learners progress from medical school to a residency program in a manner that

acknowledges each learner’s unique strengths and learning needs and optimizes professional identity formation.

The components of the transition balance the tension between individual freedoms and the public good and

provide trustworthy documentation of competence across the continuum using reliable assessment tools that

generate meaningful information for learners, educators, and where appropriate, regulators. Additionally, the

UME-GME transition is flexible and adaptable to changes in medical education and the health care system, with a

commitment to continuous quality improvement.

Key to the success of the ideal state is a commitment to the broad inclusion of students, educators, schools,

programs, and the public in the design, evaluation, and continuous improvement of the UME-GME transition.

Stakeholders are transparent and reliably provide necessary information to each other; stakeholders are trusting

and trustworthy.

Costs, nancial and otherwise, are right-sized throughout the process to maximize value, acknowledge conflicts

of interest, and allocate resources to advance the public good. Learners are prepared to serve diverse patient

populations, minimize disparities, and elevate equity as they execute the social mission of medicine and its contract

UME-GME REVIEW COMMITTEE

with the public. Diversity is present and valued throughout all specialties, programs, and geographic areas.

Appropriate action is taken to mitigate racism and harmful bias throughout the medical education and health

care systems. Faculty, learners, and the system structure cultivate inclusive learning environments that foster a

growth mindset. Medical students are provided reliable, high quality advising, and are ultimately responsible for

their own career progression after medical school.

Wellness

An ideal state for the UME-GME transition optimizes wellbeing for all involved. For learners, the nancial challenges

of applying and transitioning to residency and being a resident are minimized. Learners have adequate funding

to establish and maintain their new living arrangements and focus on their training. There is adequate but

not excessive time for the geographic move from medical school to residency. GME programs facilitate the

creation of supportive social networks for each learner with special consideration of the needs of those from

underrepresented backgrounds. A focus on health and wellbeing is integral throughout the transition.

Specialty selection

Specialty selection can be an especially fraught process for learners and impacts the eectiveness of the entire

health system. In an ideal state, medical schools have a structured approach to career advising that begins early, is

based on professional development frameworks and competencies, is integrated within an educational program,

provides broad exposure, and aligns with the needs of society. The culture of career advising programs is inclusive,

trustworthy, non-judgmental, and equitable for all students. Advising tools are high quality, interactive, honest, and

readily available.

Educators determining the structure for UME and GME programs as well as those providing advice, mentorship,

or coaching to learners recognize career indecision as a normal part of professional formation and permit

flexibility for undecided learners at key transition points. This includes allowing non-standard timelines and

nonclinical careers as necessary. Students are supported by both UME and GME to seek specialties based on

a holistic assessment of their aptitude and goals that allows learners to be aspirational about their ambitions

while pragmatic about their possibilities. This support includes access to trustworthy, data-driven resources.

Students are informed about the workforce needs of society. They are advised against contributing to a culture of

competition.

Learner selection

While learners are challenged by specialty selection, GME programs are challenged by learner selection. In an

ideal state for learner selection that benets GME programs, learners, and most importantly patients, all residency

programs receive applications from individuals with a sincere interest in attending and who are academically

prepared and aligned with the program and institutional mission. Every program receives enough applications to

ll their class and has sucient resources to conduct a holistic review of the applications received. Interviews are

oered and scheduled to promote student wellness and minimize conflict with ongoing rotations. There are ample

interview slots for those invited. Applicants interview only with programs they are likely to attend if accepted. Away

electives broaden educational exposure but are not essential for successful residency selection.

Applicants are certied by their medical school as fully prepared and trustworthy for residency training. There is

social accountability and transparency for medical schools in the validity of this certication. Residency programs

have information regarding current competence of an applicant, the trajectory of their growth during medical

school, and the accuracy of measurements. These details are available in some form for all applicants in the

Match including U.S. medical graduates (MD and DO), U.S. citizen international medical graduates (IMGs), and non-

U.S. IMGs. Programs receive early notice about any student performance concerns. These are described clearly, in

context, and with a description of the resources required for remediation or ongoing support.

UME-GME REVIEW COMMITTEE

Competence

The ideal state for learner selection requires an ideal state for the denition, assessment, and assuredness of

competence, wherein graduated medical students are prepared to serve as physicians in training. They are facile

with the appropriate knowledge, skills, and eciency and have advancing professional identity and a condent

humility. They are prepared for the realities of residency and a physician’s career. They are trustworthy to practice

under supervision, asking for help when needed.

A shared mental model of competency across the medical education continuum exists in the ideal state that

involves a standardized set of general competencies as well as specialty-focused competencies for certain

domains such as patient care and medical knowledge. Faculty development claries expectations for faculty

with learners at each level of training, teaches remediation strategies, and describes how patient safety is

ensured. Educators dene those competencies that programs believe, and data support, are the best predictors

of a learner’s abilities to succeed. Reliable and valid standardized assessment tools document competence. All

medical students engage in specialty-aligned knowledge and skills training during the nal year of medical school

to achieve the dened general and specialty-focused competencies.

Continuum and Hando

This ideal state for competence smooths a learner’s way along the continuum of medical education and allows

for seamless handos between stages. The timeline for this continuum prioritizes competence, and learners,

along with educators and institutions, approach training with a growth mindset and value lifelong learning.

Students have the time, space, and coaching to reflect on their growth and progress, grieve losses associated

with the transition to residency, and emotionally prepare for the launch of residency.

Areas for growth and gaps in a learner’s knowledge or skills are recognized and addressed by medical school

educators and GME programs as well as by themselves. Educators and learners value a learner’s competence in

identifying knowledge and skills gaps and together enact interventions for improvement. Assessment data from

the end of medical school are utilized to create an evidence-informed handover, engaging the learner in the

process and establishing directed self-learning. These data do not negatively aect a learner’s career.

Technology

An ideal UME-GME continuum is supported by useful technology that facilitates holistic review through a common,

structured format that is trustworthy and searchable. Such technology allows programs to nd applicants based

on multiple academic metrics, details of clinical and life experiences, and additional attributes. The integration of

information from schools, letter writers, and applicants allows programs to identify U.S. MD and U.S. DO students

and IMGs who will succeed at their programs. Applicants are identied by what they desire in a program, including

but not limited to a specic program, program experiences, or program mission. Evidence-based assessments are

available, meaningful, trustworthy, and presented in a useful format.

Licensing and Credentialing

The ideal state for technology in the UME-GME transition supports an ideal state for licensing and credentialing,

which is accomplished eciently for all learner groups (U.S. MD, U.S. DO, and IMGs). Varying state requirements are

addressed smoothly, creating a timely process without excessive cost. Necessary general and specialty specic

credentialing and certication are facilitated. As appropriate, an ideal state for licensing and credentialing includes

visa management.

Life Transition and Residency Launch

An ideal state for licensing and credentialling is one factor that optimizes the ideal launch of residency training.

Other factors include program directors and residency faculty who have the training, resources, infrastructure,

and perspective to approach the resident workforce as learners. Residency faculty welcome each learner as an

UME-GME REVIEW COMMITTEE

individual, knowing their strengths and weaknesses and trusting their competence appropriately. They are able

to tailor the rst months of the residency experience to the individual trainee, with appropriate supervision and

learning tools in place to facilitate success.

Additionally, residency faculty and peers recognize and mitigate bias to ensure optimal entrustment and success

for all learners in an inclusive environment. Special populations receive additional attention. This includes ensuring

that those who are underrepresented in medicine are introduced to support networks. International medical

graduates have focused training to prepare for success in the U.S.

Meaningful information about learners identied after the start of residency is also shared back to medical

schools to continually improve the preparatory process.

The ideal UME-GME transition also includes the cooperation of patients who are appropriately oriented to a clinical

environment that includes learners.

Residency Environment

Once residents start a GME program, the ideal residency environment includes adequate resources to support the

pursuit of individual learning plans for every resident.

In the ideal state, program directors and faculty have protected time, educational support, administrative sta,

professional development, and funding to support the ongoing individualized growth and wellbeing of residents.

Sponsoring institutions and all other parties recognize the primary role of resident physicians as learners and fully

support the educational environment. At the same time, the developmental path of resident physicians includes

progressive responsibility, self-directed learning, and professional identity formation, which leads to readiness for

independent practice at the time training is complete. Resources invested in medical education are appropriately

allocated to address the demands of the continuum.

Conclusion

With the successful execution of the steps of the ideal state, learners achieve an optimal transition from the role of

student to resident physician and are well prepared for the rigors of residency training.

UME-GME REVIEW COMMITTEE

Impact of Public Commentary

Stakeholder engagement has been a consistent priority for the UGRC. Ongoing updates about the Committee’s

work have been provided through the Coalition website and press releases, and through deliberate outreach and

meetings with stakeholder groups including students, program directors, DIOs, medical educators, medical school

deans, and international medical graduates. Eight of these meetings occurred after the release of the preliminary

recommendations, facilitating discussion about individual recommendations.

There have been three formal opportunities for individuals and organizations to provide feedback to the UGRC.

In December 2020, a survey was released to stakeholder organizations asking for input on the ideal state of the

UME-GME transition. Thirty-two organizations responded to the survey and the ideas they shared reinforced the

shared vision created by the UGRC for the future ideal state of the transition.

In February 2021, the UGRC issued a second call, inviting individuals and organizations to share ongoing or piloted

innovations that address concerns about the UME-GME transition. In total, 35 responses containing 39 self-

described innovations were submitted for review. Of note, the majority of the innovations submitted had previously

been identied by the Committee.

The most ambitious solicitation requested external stakeholder feedback on the UGRC preliminary

recommendations and coincided with their public release on April 26, 2021. A digital survey instrument was

created with a prominent link on the Coalition’s website. The link was made widely available to interested parties

and all stakeholder groups. The call for public comment was disseminated through numerous communication

channels including social media platforms, email distribution lists, outreach presentations, and individual networks.

In addition, periodic reminders were issued throughout the one- month open call to increase the number of

responses.

The public comment survey responses were analyzed by a team from the NBME with expertise in qualitative

and quantitative methods, and their full report can be found in Appendix E. This report, all survey comments,

and the Coalition organizational responses were made available to all UGRC members before the Committee’s

recommendations were nalized. Multiple members of the UGRC’s Executive Committee read the survey report

in full, and ve workgroup leaders were assigned to summarize and present commentary relevant to each of the

preliminary recommendations at the June 2021 UGRC meeting.

During that meeting, a comprehensive discussion about stakeholder feedback regarding the preliminary

recommendations occurred, which included 1) individual Coalition member feedback; 2) reactions from meetings

with stakeholder groups; and 3) individual and organizational responses to the public comment survey, including

statements from organizational members of the Coalition.

In response to stakeholder feedback during the public comment period, the UGRC made important changes to its

preliminary recommendations. The changes included signicant editing, clarication, and renement of language;

complete reworking of a recommendation addressing application inflation; judiciously combining similar ideas to

reduce the overall number of recommendations from 42 to 34; and sequencing of recommendations to provide

prioritization and a timeline for implementation. The feedback also helped clarify the concept of bottlenecks

or critical recommendations that must be implemented to allow other downstream recommendations to

move forward. The themes used to organize the recommendations were condensed from 12 to nine, with

reconsideration of the descriptive theme titles. Of note, 32 of the preliminary recommendations were impacted by

the feedback obtained through public commentary. Stakeholder input from individuals, stakeholder groups, and

Coalition organizations was invaluable in informing the UGRC’s nal recommendations.

UME-GME REVIEW COMMITTEE

Consolidation and Sequencing

Early feedback about the UGRC’s 42 preliminary recommendations suggested that the sheer number was

somewhat overwhelming, that some recommendations were markedly similar to each other, and that critical

recommendations might serve as upstream bottlenecks that could hinder downstream implementation of other

recommendations. To address these concerns, the co-chairs created a sixth workgroup and tasked them to

review the preliminary recommendations and determine which ones were interdependent. Once identied, these

interdependencies would serve as the basis for deciding how the nal recommendations might be organized,

consolidated, and sequenced.

This new team – known as the “bundling workgroup” – considered the preliminary recommendations for the

purpose of consolidation and reducing their total number. The workgroup also reviewed feedback from the public

commentary to decide which recommendations could be grouped together, and in what sequence, to facilitate

orderly and ecient implementation. This work led to the identication of four distinct bottlenecks: organizational

collaboration and continuous quality improvement (Recommendation 1), the creation of an interactive database

(Recommendation 6), developing consensus around a common outcomes framework (Recommendation 9), and

away rotations (Recommendation 13).

The gure depicts how Recommendation 9 acts as a

bottleneck. Specically, Recommendation 9 calls out

the need for a common outcomes framework shared

by both UME and GME. If no consensus is achieved on a

common outcomes framework, progress in implementing

the following three recommendations will be impeded:

Recommendation 11 (related to assessment tools associated

with the common framework); Recommendation 12

(related to faculty development for both teaching and

assessment to optimally utilize the common framework),

and Recommendation 20 (related to an electronic

dashboard that shows each learner’s assessment data

within the framework).

Although each bottleneck is associated with a number of downstream recommendations, the most important

bottleneck is Recommendation 1, which calls out the need to convene a national ongoing committee to manage

continuous quality improvement of the entire process of the UME-GME transition. As shown in the table below, 27

of the nal 34 recommendations depend on the implementation of Recommendation 1. Without organizational

collaboration to convene a national committee focused on continuous quality improvement, the UGRC

recommendations for comprehensive improvement of the UME-GME transition will fail for lack of implementation.

UME-GME REVIEW COMMITTEE

Bottleneck Downstream, Dependent Recommendations for each Identied Bottleneck

1 2 3 4 5 6 7 8 11 12 14 15 16 17 18 19 20 23 24 25 26 28 29 30 31 32 33 34

6 7 8 16 23 24

9 11 12 20

13 23 24

As noted above, the bundling workgroup’s discussions were informed by feedback obtained from the public

comments. However, the tasks of organizing and sequencing the recommendations could not be nished until

the UGRC had adopted nal recommendations. After the public comment period closed and the analysis of the

commentary was completed, most of the preliminary recommendations were modied and one was completely

reworked. The workgroup then proposed that the Committee consolidate 13 preliminary recommendations

into ve, new bundled recommendations. The UGRC accepted this proposal, which reduced the number of

nal recommendations from 42 to 34. Further, the UGRC adopted the workgroup’s proposal to re-organize the

recommendations into the following nine themes:

• Collaboration and Continuous Quality Improvement

• Diversity, Equity, and Inclusion

• Trustworthy Advising and Denitive Resources

• Outcome Framework and Assessment Processes

• Away Rotations

• Equitable Mission-Driven Application Review

• Optimizing Application, Interview, and Selection Processes

• Educational Continuity and Resident Readiness

• Health and Wellness

Lastly, the UGRC adopted the bundling workgroup’s proposal on sequencing the nal recommendations within

each theme. This organizational structure reflects the interdependence of the recommendations and is intended

to help stakeholders, including the organizational members of the Coalition, consider next steps. As shown in the

table on the next page, each recommendation has a proposed initial timeframe to help guide its implementation.

Note that the UGRC considers certain recommendations (i.e., numbers 21 and 22) to be time sensitive and that

these should be enacted immediately by the Coalition. In contrast, the Committee members understand that

others may require several years of development before they can be fully implemented.

UME-GME REVIEW COMMITTEE

Sequencing Timeline

July-21 2021-22 2022-23 2023-24 2024+

Now Immediate Soon Longer

Theme Collaboration and Continuous Quality Improvement

1 Committee to Manage CQI across Transition

2 Residency Selection and Physician Workforce Research

3 IRP Reform

Theme Diversity, Equity and Inclusion

4 Specialty Specic Practices to Increase Diversity

5 DEI Education Across the Continuum

Theme Trustworthy Advising and Denitive Resources

6 Interactive GME Database

7 Career Advising Resources

8 Career Advising Curriculum

Theme Outcome Framework and Assessment Processes

9 Common Competencies across Transition

10 CQI to Mitigate Bias across Transition

11 Improved Assessment Tools

12 Competency Based Faculty Development Materials

Theme Away Rotations

13 Review of Away Rotations

Theme Equitable Mission-Driven Application Review

14 MSPE Revision

15 Structured Evaluative Letters

16 Sharing Applicant Demographics to Improve Diversity

17 Electronic Application System Improvements

18 Reporting Licensure Exams in a Single Field

19 Review of Filter Content and Use

20 Standardized Dashboard and Portfolio for Learners

Theme Optimizing Application, Interview and Selection Processes

21 Virtual Interviews for 2021-2022

22 Standards for Interview Oer and Acceptance Process

23 Residency Application Process Innovations

24 Interview Limits

Theme Educational Community and Resident Readiness

25 Feedback from GME to UME

26 Centralized Resident Support Resources

27 Coaching for Professional Identity Formation

28 Specialty Specic Residency Preparation

29 Improved Residency Program Orientation

30 UME to GME ILP Hando

Theme Health and Wellness

31 Wellness Resources for the Transition

32 Assured Time Between UME and GME

33 Equitable Access to Funding for Transition

34 Standardized Process for Initial Licensing

UME-GME REVIEW COMMITTEE

Limitations

The UGRC faced a number of limitations to its work process, beginning with the allotted timeframe. Although the

deadline for creation of the nal recommendations was 10 months, the complexity of the problems intrinsic to the

current UME-GME transition and the comprehensive scope of the charge were considerable.

In addition, a signicant amount of time was devoted to the steps prior to generating recommendations, including

the elaboration of the charge, level setting, envisioning an ideal future state, and generating Ishikawa diagrams

(shbone analyses) of root causes of the identied challenges to the transition. In the end, this eort was thought

necessary, as the process goal was to articulate the desired outcomes and understand the root problems before

generating solutions.

Due to time constraints it was also necessary to divide the charge between workgroups, and have work proceed

in parallel. More opportunities to share information and explore revisions across workgroups may have allowed for

earlier consolidation of recommendations.

Although the virtual format of the meetings caused some limitations with regards to the social norms of in person

meetings, the ability to come together virtually had a positive impact on the eciency of the work process.

The chat function of the platform also enabled additional layers of interactions and an alternative modality of

engagement. Managing a committee of 30 members can be challenging, but UGRC meetings were structured to

include both small and large group sessions.

Although the UGRC had representation from across the continuum, there was an acknowledged desire for more

diversity within the Committee and more representation from learners.

The workgroups and the librarians searched for medical literature supporting proposed solutions to the problems

facing the transition, however there is currently limited evidence for many of the recommendations, and

generating future research questions became an additional focus of the Committee’s work.

Although many innovations to improve the UME-GME transition are being explored across the country, it is too early

to draw conclusions regarding their overall eectiveness. As evidence accrues, a continuous quality improvement

process will help to advocate for changes that are evidence based.

The number of comments received through the public comment survey was lower than expected. This was likely

secondary to the large number of preliminary recommendations presented for feedback, and there was ongoing

discussion on how to most eectively engage stakeholders. Received survey comments overall however were

thoughtful and high quality, and signicantly impacted the wording of the nal recommendations.

Although there was the potential for conflict of interest from the inclusion of organizational leadership on the

UGRC, members honored the explicit request to bring their experience and expertise to discussions, and to

participate as individual members rather than as organizational representatives.

The scope of the UGRC was limited to the UME-GME transition, and therefore recommendation timeframes do

not extend for the duration of residency or into fellowship training. The resources available to the UGRC also did not

allow for a cost analysis of the recommendations.

Finally, the charge for the UGRC was to generate solutions in the form of recommendations to comprehensively

improve the UME-GME transition. We look to the collaboration of the Coalition member organizations for

implementation of the recommendations.

UME-GME REVIEW COMMITTEE

References

1. Whelan A, Joshi A. Planning Committee for the UME-GME Review Committee Final Report. Coalition for Physician

Accountability Planning Committee. 2020. https://physicianaccountability.org/wp-content/uploads/2020/05/

Planning-Committee-Final-Report-Draft-Final_520.pdf (Accessed June 30, 2021).

2. Barone MA, Filak AT, Johnson D, Skochelak S, and Whelan A. Summary Report and Preliminary

Recommendations from the Invitational Conference on USMLE Scoring (InCUS), March 11-12, 2019. https://www.

usmle.org/pdfs/incus/incus_summary_report.pdf (Accessed June 30, 2021)

3. Cain R, Catanese V, Katsufrakis P, McMahon G, Odom C, and Skochelak S. Proposal: Planning process to

collaboratively review the transition from UME to GME. Coalition for Physician Accountability Management

Committee. 2019. https://physicianaccountability.org/wp-content/uploads/2020/01/UME-GME-Proposal-

Final-1.17.20.pdf (Accessed June 30, 2021)

4. Chaudhry HJ, Kirch DG, Nasca TJ, et al. Navigating Tumultuous Change in the Medical Profession: The Coalition for

Physician Accountability. Acad Med. 2019 Aug; 94(8):1103-1107.

UME-GME REVIEW COMMITTEE

APPENDICES

UME-GME REVIEW COMMITTEE

Appendix A: Glossary of Terms and

Abbreviations

Organizations:

ACCME: Accreditation Council for Continuing Medical Education

ACGME: Accreditation Council for Graduate Medical Education

AACOM: American Association of Colleges of Osteopathic Medicine

ABMS: American Board of Medical Specialties

AMA: American Medical Association

AOA: American Osteopathic Association

AAMC: Association of American Medical Colleges

CMS: Centers for Medicare and Medicaid Services

CMSS: Council of Medical Specialty Societies

COALITION: Coalition for Physician Accountability

ECFMG: Educational Commission for Foreign Medical Graduates

FSMB: Federation of State Medical Boards

LCME: Liaison Committee on Medical Education

NBME: National Board of Medical Examiners

NBOME: National Board of Osteopathic Medical Examiners

NRMP: National Resident Matching Program

OPDA: Organization of Program Director Associations

UGRC: Undergraduate Medical Education to Graduate Medical Education Review Committee

Terms:

Away Rotations: A clinical experience at a teaching hospital or clinic that is not aliated with a student’s medical

school

Basic Advising: Common understanding of career advising, professional development, specialty selection, and

application procedures.

CiM: Careers in Medicine

COMLEX-USA: Comprehensive Osteopathic Medical Licensing Examination of the United States

Competence: The array of abilities (knowledge, skills, and attitudes) across multiple domains or aspects of

performance in a certain context. Statements about competence require descriptive qualiers to dene the

relevant abilities, context, and stage of training. Competence is multi-dimensional and dynamic. It changes with

time, experience, and setting. (Frank et al. 2010)

Competency: An observable ability of a health professional related to a specic activity that integrates knowledge,

skills, values, and attitudes. Since competencies are observable, they can be measured and assessed to ensure

their acquisition. Competencies can be assembled like building blocks to facilitate progressive development. (Frank

et al. 2010)

CQI: Continuous Quality Improvement

DEI: Diversity, equity and inclusion

DIOs: Designated institutional ocials

UME-GME REVIEW COMMITTEE

DO: Doctor of osteopathic medicine

Dual Applicants: Applicants applying to more than one specialty

Educational Continuum: Term that describes the span of a physician’s education, from undergraduate medical

education (medical school) to graduate medical education (residency and fellowship) to continuing medical

education (ongoing during years in practice)

ERAS: Electronic Residency Application Service

General Career Advising: Assisting students in selecting an appropriate career path and specialty

GME: Graduate medical education (residency training)

Holistic Review: Mission aligned selection process that considers an applicant’s experiences, attributes, and metrics

ILP: Individualized learning plan

IMG: International medical graduate

InCUS: Invitational Conference on USMLE Scoring

Initial Residency Period (IRP): Number of years it takes for a resident to become board eligible in the rst medical

specialty the resident entered, set when a physician enters residency

In-Training Examination: Annual specialty-specic standardized multiple choice question medical knowledge

examination

Longitudinal Assessment: Measurement of a learner’s knowledge, skills and attitudes that occur on an ongoing

basis over a prolonged period of time

LOR: Letters of recommendation

Matched: A student that was able to secure a residency position to continue their medical education

Match Day: The day, typically occurring in March of each year, that students nd out which residency program

they have been assigned to for training after they graduate from medical school

MD: Doctor of medicine

MSPE: Medical Student Performance Evaluation

PCP: Primary care provider

PD: Program director

SELs: Structured evaluative letters

SOAP: Supplemental Oer and Acceptance Program

Specialty-Specic: Pertaining to a specic medical specialty (e.g., pediatrics, surgery, psychiatry, etc.)

Specialty-Specic Advising: Assisting students with strategies for optimal placement in their chosen specialty

SLOEs: Standardized letters of evaluation

Un-Matched: A student that was unable to secure a residency position to continue their medical education

USMLE: United States Medical Licensing Examination

Lack of alignment:

advising & stakeholder

needs

• Advising misaligned

with student

preferences

• Advising not aligned

with patient &

population health needs

Student advising Assessment tools and strategies Culture

Lack of trustworthy data to

inform advising

• Lack of transparency —

programs don’t share all data

• Data not standardized across

schools and programs

Inadequate advisor preparation

• Lack of time

• Lack of funding

• Lack of institutional value placed

on advising

• Lack of current advising resources

Lack of single coordinated system

• Health care nancing uncoordinated

• System needs things that don’t have a good business model

Each stakeholder has own nancial interest

• Separate funding streams

• Rigid business model in US and international schools

Each stakeholder has own accountability

structure

• All stakeholders are looking at dierent part of

the problem

• All stakeholders are working in their own interest

Public as stakeholder is undervalued

• Metrics of success focus on learners,

schools & programs

• Accreditation is process-oriented

rather than patient outcome-focused

Stakeholders Match system Denition of competence

Lack of shared mental model

• Purpose of assessment; tension of

formative vs. summative

Varied approaches to assessment

at schools

• Culture of individual ownership of

approaches

• Lack of trust in available tools and

strategies, users

Varied, insucient resources for

assessment

Lack of validity evidence for

assessment tools and strategies

• Lack of expertise to generate

evidence

• Inadequate resources

Transactional, high-stakes nature

of match disconnected from

educational priorities

• PDs are focused on comparative

data rather than competence

— Inadequate time to review

applications

• Pressure to achieve high rankings

and be perceived as excellent

using familiar metrics

— Don’t have a way to measure

some important outcomes

Rigid, one size ts all approach

• Fixed timepoint for match

• Implicit assumption that all

learners will progress at equal pace

Culture is competitive

• Students are competitive to get into

medical school

• Residency selection is competitive

• Schools are competitive

• Lack of trust in other stakeholders

• Focus on individual achievement

over social good

• Insucient or ineective attention

to professional identity formation in

training

Challenges of labeling

unprofessional behavior

• Fear about impact of labeling

unprofessional behavior

• Lack of understanding of

professional development

• Generational Dierences

• Legal risks

Fear of failure

• Risk of unmatched student

• Culture success

Variable denition of competence

• Competence has local meaning

in the local context & culture

• Variable facility development

• Competence as achievement vs.

a developmental progression

• Dierent denitions of

competence in UME & GME

Variable value placed on

competence

• Variable faculty and institutional

buy-in to CBME

• CBME doesn’t provide maximally

useful info in the Match

Misaligned incentives

• Easier to advance a learner than

stop them from advancing to

ensure competence

The current

dysfunctional UME-GME

transition system is

characterized by

mistrust and mismatch

of expectations among

learners, UME educators,

and GME educators who

use the wrong

information to make

wrong inferences

••••••••••••••••

Workgroup A

Appendix B: Workgroup Ishikawa Diagrams (Fishbones) Created for Root Cause Analysis

Key for anticipated time-based solutions: short term solution mid term solution long term solution

Applicants clinical

experience and

knowledge of specialty

limited

Applications

Recruitment

Application Mechanics: Lack of

clarity with reporting required

information (LOA, probation, etc.),

costly, long application season,

couples match process dicult to

understand

Letters of recommendation lack

consistency among specialty requirements

with confusion/bias to templates

Interviews lack standardization across

programs and specialties, lack faculty

development and education on

appropriate questioning

Virtual interviewing:

• Bias to tech issues, perception of

staging, and resource cost

• Tech limitations due to cost,

availabilities and tech diculties

Student interviewing-

Preparation resource intensive

Cost- of interview high in both

tangible costs (travel) and time

away from rotations

Interviews

Assessment and

Holistic Review

Medical student reporting lacks consistent,

comparable information from objective and

universal reporting tools leading to mistrust by

residency programs

Application inflation increasing

program reliance on board scores

Programs lack consistent data and

resources for holistic review of

applicants

Medical school advising challenges

due to lack of feedback from

programs to school, advisor

experience, variation/lack of

program data, variabilities in

student clinical exposure, lack of

clarity on application requirements

Chaotic Interview Process

• More interview oers than slots

resulting in cancellation

• Insucient opportunity to

respond to interview invitation

• Holding interview slots

Lack of real-time interview update to

allow schools to intervene/counsel

students appropriately

Medical school reliance on

high residency match rates

for their own recruitment

may conflict with GME goals

The system of scheduling elective rotations

is problematic for programs (duplicative

scheduling of rotations, cancellations)

EPAs and other competency reporting requirements

vary among schools and lack standardized tools that

are measurable, reproducible, universal, and

understood by students

Lack of professionalism information as

denitions, faculty development and tools

for reporting vary among schools

Inflexibility of med school

curriculum to tailor educational

experiences for successful

residency transition

The UME/GME

process for the

applicant, medical

school, and program

requires improvement

•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••

The system lacks transparency and visibility of

program requirements for away or audition

rotations, including if these ‘lters’ dier from

interview/residency application lters

The system lacks an easily accessible

database for programmatic information

that is accurate and comparable that

includes the program’s desired applicant,

including ‘lters’ used by program to

determine student eligibility

Lack metrics in multiple key competencies

to compare candidates leading to program

reliability on board scores

Day 1 Readiness/Later Transition

lacks denition and consistent tools

for measurement and reporting

For all 4 components-there

is a lack of consistent

denition and application

of DEI practices

•••••••••••••

Workgroup B

Fear

Lack of Trustworthy,

Validated Information

to Programs

Lack of Trustworthy,

Validated Information

to Applicants

Needs of Society

Not Prioritized

Program Director

Stress (Expectations)

Program Director Stress

(Limited Resources)

Applicant Stress Bias

Lack of trustworthy assessment

especially with respect to

longitudinal, workplace-based

and 360 degree assessment,

including for IMGs

Applicant information is not in a

structured, validated format

usable for large scale review

Lack of understandable, plain

language reporting of student

assessment pre- and post-match,

especially in longitudinal,

workplace based and 360 degree

assessments, especially for IMGs

Unclear what data should be used

for resident selection, or how we

would dene a successful resident

Fear of missed opportunity-

PDs want “best” applicants,

applicants want “best”

programs, both think that

one more interview or

application will help

UME-GME

transition process

is inequitable,

inecient,

wasteful, costly,

and unnecessarily

stressful for all

involved

•••••••••••••••••••••

Fear of not lling- pressure for

programs to ll due to funding,

clinical need, prestige, etc.

Fear of not matching- Applicants

have limited career options outside of

the Match and perceive no flexibility

to change specialty or the timeline

Medical schools fear unmatched

students, may limit their

transparency

Inflexible timeline

Unfamiliar process

ACGME requirements to address wellness,

QI, diversity, and board pass rates

increase documentation and stress

Frequent Program director turnover, so new

PDs must learn unfamiliar rules

Hospital partners have clinical

expectations for the program without

a lot of backup if the program can’t

meet those expectations, which leads

to signicant risk aversion (for learners

who could struggle) and fear of not

lling (which also aects program

funding)

There are many applications per

position, and many applicants

have similar qualications. PDs

have little guidance on how to

select applicants for interview as a

part of holistic review

Limited time and stang for

individual holistic review at initial

application review, so may rely

on simplistic lters

Limited funding may fall

further if program begins to

struggle and can’t ll

Inadequate resources for

trainees requiring additional

support (educational to pass

boards, psych, clinical backup

if unable to care for patients,

faculty development, etc.).

Lack of resources means that

learners who needed support

previously are avoided

Limited time and stang

for interviews

Program director burnout and

depression may lower their

capacity further

Financial burden, educational

burden, and opportunity cost

for time spent on application

process

Unfamiliarity with the

process

Obligation for away

electives for more than

broadening clinical diversity

or learning about a

program– some specialties

required them for interest

signaling and student

assessment

Process is very dierent for

dierent groups of applicants

(USMD, USDO, IMG, etc.),

without clear expectations

Any measurement technique

(including standardized

metrics) can hinder some

applicants, but programs

need some way to tell the

dierence among them, and

applicants want a way to

distinguish themselves

Using biased metrics for

selection leads to a more

transparent, predictable

process compared with

holistic review

Filters can cause bias without

alerting programs (ie USMLE

lters removing DO applicants)

Bias favors certain applicants,

schools, etc., who may resist

complete equity

Conflicting advice from

multiple sources (peers, UME,

GME, online)

Yearly variability in residents

matched, especially at

smaller programs

Programs are not always

transparent in how they

select applicants for

interview and ranking, or

who actually matches with

the program

Applicants do not seem

to utilize or trust the

information that is

available.

Student eort spent on transition

instead of working toward the greater

good (research, patient care, wellness)

Students learn to hide their

weaknesses, reinforcing unhelpful

patterns for future practice

Sucient applicants do not go to

underserved areas/specialties

(FM, IM, and peds are the most

unlled specialties)

Learner-centered educational

requirements for residency programs

may conflict with patient-centered

health system needs

DO, URM, and IMG applicants are

underrepresented in certain

specialties and geographic areas

Signicant waste due to

redundant licensing exams

(multiple steps of both COMLEX

and USMLE, some applicants take

both). Uncertain that these metrics

are predictive of competence.

Workgroup C

Professional Identity

Formation

Information Sharing -

Handos

Optimizing UME:

Residency Ready

Optimizing GME:

Ensuring Patient Safety

DEI

Logistics of

Transitioning

Feedback to UME

Wellbeing

Learning Environment

Cultural Dierences | Mistreatment | Hidden Curriculum | Service/Education Balance | Specialty Disrespect

Clinical Environment

Patient Safety | Quality | Wellbeing | Professionalism | Supervision and Care Transition

Individual

Attributes

Diversity

URM

IMG

MD/DO

Support

Partners

Parenting

Finances

Post Match

Optimization

Eect

•••••••••

Pressure for schools

to advocate for

applicants

Meaningful data, ILPs

FERPA

No consequence to

UME for inaccuracy

Discipline-specic

curriculum linked to GME

Few non-residency

options

Conflict of interest

for schools

CoreEPAs

Boot camps —

Skill building

Lack of flexibility to

make customized

schedules

Appropriate supervision

Bd pass rate as

measure of success

Cultural humility

Lack of early

assessment

Low risk capacity —

Challenge of the

struggling intern

Importance of IPE

Need for representative,

inclusive community

Need for bystander

training

Need for equity lens

throughout process

Mindset transition:

student to worker &

lifelong learner

Life transitions

Impact of

mistreatment

Imposter

syndrome

Lack of

coaching

Graduation

Life long

learning

Specialty

bias

Moving

Finances

Licensure

Varied state

requirements

GME performance

data

Visa issues

Curricular gaps

Trainings

(ACLS, etc.)

MSPE utility &

accuracy

Welcome &

support

Need for early

assessment

Workgroup D

UME-GME REVIEW COMMITTEE

Appendix C: UGRC Final Recommendations

With Complete Templates

Recommendation 1:

Convene a national ongoing committee to manage continuous quality improvement of the entire process

of the UME-GME transition, including an evaluation of the intended and unintended impact of implemented

recommendations.

Narrative description of recommendation:

One of the challenges in creating alignment and making improvements is the lack of a single body with

broad perspective over the entire continuum. This creates a situation where organizations and institutions

are unnecessarily and counterproductively isolated, without a shared mental model or mission. A convened

committee, that includes learner and public representatives, should champion continuous improvement to the

UME-GME transition, with the focus on the public good.

This recommendation creates the ideal state for the UME-GME transition because:

The ideal state requires an equitable, coordinated, ecient, and transparent system across the UME-GME

transition. Further, the ideal state specically endorses the idea that the transition ecosystem must adapt to

changes in both medical education and health care, with a commitment to continuous quality improvement.

An ongoing committee that is focused on the entire process will ensure that the eorts to implement all

recommendations occur in a coordinated fashion, and that sucient attention is given to doing so in a manner

that is committed to continuous quality improvement.

How this recommendation links to the shbone diagrams used to develop the ideal state:

• Much of the UME-GME transition occurs within clinical learning environments

• Inequitable, inecient, wasteful, costly, and unnecessarily stressful

Implementation “must haves” include:

• Buy in from each constituency to allow for eective launch and operations

• Commitment to continue the work of implementing the UGRC recommendations

• Continuous quality improvement mindset

Implementation “nice to haves” include:

• Benchmarking and dashboards to show progress toward implementation

• Communications vehicles to disseminate future recommendations

Pros Cons

Alignment Reluctance to support central authority

Cost savings

Creation of a new organization or expansion of a

current one with associated costs and turf/scope/

boundary challenges

Focus on the public good Risk of increasing expense or hassle

Iterative and evolving

UME-GME REVIEW COMMITTEE

Specic examples on how this recommendation might be implemented:

• The Coalition for Physician Accountability is itself an example of an overarching organization that has self-

organized into a body issuing recommendations on numerous topics. Similarly, a subset of organizations

(i.e., American Association of Colleges of Osteopathic Medicine, Association of American Medical Colleges,

Accreditation Council for Graduate Medical Education, Educational Commission for Foreign Medical Graduates/

Foundation for Advancement of International Medical Education and Research) has recently provided resources

to help graduates from the Class of 2021 make a successful transition from students to residents. Further,

various organizations have collaborated on a number of projects such as professionalism (American Board of

Medical Specialties, National Board of Medical Examiners) and USMLE Scoring (Association of American Medical

Colleges, American Medical Association, Educational Commission for Foreign Medical Graduates, Federation of

State Medical Boards, and National Board of Medical Examiners). Another example is the creation of steering

committees that include individuals to provide guidance of specic initiatives (e.g., the AMA’s Accelerating Change

in Medical Education initiative). The Coalition for Physician Accountability could create a permanent committee

or group to launch implementation, provide ongoing guidance, monitor progress, and recommend future action

based on data and environmental factors.

• A subset of the Coalition for Physician Accountability (i.e., those organizations with the most interest in the UME-

GME transition) could self-organize to create and support a central oversight body.

Research questions:

• Apart from accrediting agencies, are there descriptions in the health professions, law or business literature of

ongoing, permanent oversight bodies that have been created to oversee the transition of professionals from

learners to practitioners?

• What is the structure of such oversight bodies and is there any evidence that they have operationalized a

continuous quality improvement mechanism?

• Are there examples of collaboration or public-private partnerships (e.g., government agencies and NGOs) that

have successfully implemented oversight over a professional transition?

UME-GME REVIEW COMMITTEE

Recommendation 2:

In addition to supporting collaboration around the UME-GME transition, this national committee should: develop

and articulate consensus around the components of a successful residency selection cycle; explore the growing

number of unmatched physicians in the context of a national physician shortage; and foster future research to

understand which factors are most likely to translate into physicians who fulll the physician workforce needs of

the public.

Narrative description of recommendation:

Currently, the medical education community lacks a shared mental model of what constitutes a successful

transition from UME to GME, and also what factors predict that success. The lack of agreement leads to conflict

over the content of applications as well as the resources required for a residency selection cycle. Success could

include simple educational outcomes such as completing training, board certication, or lack of remediation.

Alternatively, applicant-specic factors may be more important, such as likelihood of choosing the same program

again. Success may be dened solely on the public good, based on the ll rate of programs and the number

of physicians practicing in underserved areas. Or, it may be that successful residency selection is institutionally

specic based on its mission and community served, with some institutions focused on research and others on

rural communities. The committee should articulate the factors associated with a successful residency selection

cycle so they can be appropriately emphasized in the UME-GME transition, especially as changes are made to the

process.