If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
The Accreditation Council for Graduate Medical Education mandates that residencies have strategies in place to recruit applicants who are historically underrepresented in medicine.
•
Our national survey of internal medicine residency program directors revealed a range of approaches for recruitment with varying degrees of perceived effectiveness.
•
Barriers include the lack of diverse faculty and insufficient institutional support.
•
This collection of strategies can inform residency initiatives to bring diversity to the physician workforce.
Introduction
Racial and ethnic concordance between physicians and patients can improve health care delivery.
However, the US physician workforce does not adequately reflect the demographic composition of the US population: as of 2018, only 17.1% of physicians self-reported as Asian, 5.8% Hispanic, and 5% Black or African American.
Such disparities within internal medicine have broad health and societal repercussions, because internists and other generalists represent frontline providers to the majority of the population seeking health care.
The Accreditation Council for Graduate Medical Education (ACGME), among other regulatory bodies, has embraced the value that diversity brings to the culture and practice of medicine. In 2019, it mandated that residency and fellowship programs improve the diversity of their workforce.
However, programs were not provided with guidelines or resources for improving recruitment of historically underrepresented in medicine (URiM) applicants, resulting in heterogeneous efforts of unclear value.
Several studies have demonstrated the effectiveness of single-institution interventions such as externships and second visits,
To our knowledge, a broad-based undertaking has not been conducted of how residency programs recruit URiM applicants and whether these efforts are viewed as effective.
In the face of a national mandate for improving diversity, programs are developing their own solutions without being informed by what others are doing successfully. To aggregate URiM recruitment strategies among a large sector of residency programs, we used a nationally representative survey of internal medicine program directors to solicit strategies, their perceived effectiveness, and other barriers to recruitment.
Methods
Study Setting and Participants
The Association of Program Directors in Internal Medicine (APDIM) is a charter organization of the Alliance for Academic Internal Medicine (AAIM), a professional association that represents over 10,000 internal medicine educators and administrators. The APDIM Survey Committee oversees the development of an annual research survey of internal medicine residency program directors to study critical issues in graduate medical education training. Thematic survey sections vary annually, although a “core” set of questions about program characteristics remains static. The 2019 survey was disseminated to program directors at all 422 APDIM member residency programs with ACGME accreditation prior to July 1, 2018. At the time of the study, APDIM member programs represented 82% of ACGME-accredited residency programs.
Instrument Design
In May 2018, a call for thematic survey proposal submissions was disseminated online to all APDIM physician-members (approximately 4500). In October 2018, the APDIM Survey Committee blind-reviewed question section proposals, scored them on merit and relevance, and selected 5 sections for inclusion in the 2019 survey. The proposal authors adapted the survey instrument used in previous work
based on its findings and submitted questions for consideration. The section “Recruitment Strategies for Underrepresented in Medicine Applicants to Internal Medicine Residency” was selected. It consisted of 39 questions with conditional (skip or display) logic. Questions solicited program emphasis on URiM recruitment, access to diversity offices, recruitment strategies along with perceived effectiveness, impact of ACGME requirements on recruitment efforts, and barriers to recruitment. Response types included multiple-choice, Likert-type questions, and write-in fields for responses of “other.” The survey section ended with an open-ended question: “What is the one area you wish your institution would focus on to recruit URiM trainees?”
From February through May 2019, the questions were edited and revised by the committee in consultation with section coauthors; in June, AAIM staff programmed the instrument in the Qualtrics Surveys platform in preparation for committee pretesting, author revision, and pilot testing. The study (#18-AAIM-107) was deemed exempt by Pearl IRB (Indianapolis, Ind; US Department of Health and Human Services #IRB00007772) in accordance with Food and Drug Administration 21 Code of Federal Regulations 56.104 and Department of Health and Human Services 45 Code of Federal Regulations 46.104. The survey launched on August 12 and closed on December 9, 2019, and included 5 e-mail reminder messages to nonrespondents.
Statistical Analysis
Quantitative data analysis was conducted in Stata 16 SE (College Station, Texas). Prior to de-identifying the final responses for analysis, the study population dataset was appended with data from external sources, including US Census Bureau geographic region.
Residency program characteristics that explained most of the population variance, such as number of approved resident positions, were obtained from ACGME.
American Medical Association. Fellowship and residency electronic interactive database access system online. Available at: https://freida.ama-assn.org/. Accessed October 3, 2021.
Descriptive statistics for analysis included the reporting of frequencies and percentages for categorical variables and measures of central tendency (eg, mean, median, standard deviation) for continuous variables. After review, several responses that coincided with existing response categories were reassigned accordingly; responses that did not fit existing response options remained coded as “other.” To describe the statistical representativeness of the survey responses, we compared characteristics that explained most population variance of respondents and their programs to non-respondents using the Adjusted Wald (Pearson) test of association with 1 degree of freedom for categorical variables. To compare the means or medians of continuous variables to dichotomous variables, we used Welch's t test, an interquartile range test (Welch's t), or a nonparametric equality-of-medians test. To confirm the construct validity of self-reported items that were not mutually exclusive, we reported Cronbach's alpha (α) with average inter-item covariance. Statistical significance was designated using an alpha level set to P ≤ .05, and construct validity for Cronbach's α was deemed acceptable at 0.70 or higher.
We conducted a content analysis of open-ended responses, with 2 authors (MM, GH) reviewing all comments independently, discussing jointly, and arriving at consensus on overarching themes.
Results
The survey response rate was 69% (293 of 422 survey-eligible program directors). Among responding program directors, 104 (36%) were situated at university-based programs and 267 (91%) represented programs whose ACGME accreditation status was “continued.” The geographic distribution of programs was most prominent from the Northeast (30%) and the South (35%). The median size of programs (approved ACGME resident positions) among respondents was 52 (SD 41.9). There was no significant difference in characteristics between responders and non-responders, although there were slight differences based on the proportions of international medical graduate (IMG) trainees (P = .161) (Table 1).
Table 1Core Characteristics of Responding and Nonresponding Internal Medicine Residency Programs: 2019 Survey of US Internal Medicine Residency Program Directors
Equality-of-medians test (continuity corrected Pearson chi-squared).
52 (41.9)
48 (37.5)
50 (40.8)
.229
ABIM pass rate 2016-2018 (%); n = 252, n = 111
91.1 (6.9)
91.1 (7.1)
91.1 (6.9)
.965
Program director tenure as of 2019 (years; ACGME)
5.7 (5.8)
6.2 (6.2)
5.8 (5.9)
.454
Program accreditation year (ACGME)
1976.5 (23.3)
1977.9 (25.0)
1977.0 (23.8)
.586
ABIM = American Board of Internal Medicine; ACGME = Accreditation Council for Graduate Medical Education; AMA-FREIDA = American Medical Association Residency and Fellowship Database; IMG = international medical graduate; SD = standard deviation; VA = Veterans Affairs.
Bivariate (Adjusted Wald [Pearson] test of association with 1 degree of freedom) used for categorical variables: alpha = 0.05.
† Excludes programs from 2 US territories, due to small cell sizes/data confidentiality.
‡ Welch's t test.
§ Interquartile range test (Q3-Q1): 79.5-6.0; 85.3-22.4; 81.5-7.9.
‖ Equality-of-medians test (continuity corrected Pearson chi-squared).
Respondents reported a range of perceptions of the emphasis on URiM recruitment in their programs, with 143/293 (49%) reporting a “good” or “great deal of emphasis” and 46/293 (16%) reporting little to no emphasis on these efforts. Three-fourths (213/293, 73%) of respondents reported that their programs had access to a dedicated office or director of diversity, and the most commonly reported levels of accessible support were at the medical school (126/213, 59%) and the hospital network or health system (90/213, 42%). In response to whether they actually received support from dedicated offices or directors of diversity, 40% (51/126) of respondents with medical school diversity offices reported receiving no support, and 38% (34/90) of respondents with hospital network or health system diversity offices reported no support. In contrast, among respondents who reported to have a residency program diversity office, 57% (31/54) reported a great deal of support from the graduate medical education level (Figure).
FigureLevel of access to a designated office or director of diversity and level of supposed provided in recruiting underrepresented in medicine applicants (URiMs) to resident programs (n = 213).
We found that program directors used a variety of recruitment strategies, ranging widely in their opinions about relative effectiveness (Table 2). The most common strategies were websites (169/212, 80%), demonstrating a commitment to diversity on interview day (165/215, 77%), URiM residents and faculty being present on interview day (160/211, 76%), and matching URiM faculty to applicants (128/199, 64%). The use of race/ethnicity data in the Electronic Residency Application System (ERAS; 158/215, 74%) was also prominent.
Table 2URIM Recruitment Strategies By Usage and Perceived Effectiveness
Cronbach's alpha (α) for items reported: 0.81; average interitem covariance: 0.008.
Very Effective
Somewhat Effective
Not at All Effective
A webpage/website showcasing diversity
212
169 (79.7)
40 (23.7)
114 (67.5)
15 (8.9)
Feature diversity and inclusion as a key topic for all applicants on interview day (as well as rotations, research, academic work)
215
165 (76.7)
48 (29.1)
103 (62.4)
14 (8.5)
Feature URiMs (eg, residents, faculty) prominently on interview day
211
160 (75.8)
55 (34.4)
96 (60.0)
9 (5.6)
Use race/ethnicity data in ERAS to increase interview invites to URiM candidates
215
158 (73.5)
53 (33.5)
91 (57.6)
14 (8.9)
Match URiM applicants with URiM faculty on interview day
199
128 (64.3)
32 (25.0)
83 (64.8)
13 (10.2)
Distribute print-based materials
180
96 (53.3)
7 (7.3)
57 (59.4)
32 (33.3)
Recruit applicants at URiM association or society events
190
83 (43.7)
12 (14.5)
59 (71.1)
12 (14.5)
A special meeting with program leadership for URiM applicants prior to interview day
189
59 (31.2)
7 (11.9)
30 (50.8)
22 (37.3)
Offer a second visit for URiMs only
183
51 (27.9)
12 (23.5)
20 (39.2)
19 (37.3)
A special meeting with program leadership for URiM applicants on interview day
180
53 (29.4)
5 (9.4)
28 (52.8)
20 (37.7)
Make individual phone calls after interview day to URiM applicants only
180
40 (22.2)
6 (15.0)
12 (30.0)
22 (55.0)
Hold a separate event for URiMs during their initial visit
185
37 (20.0)
7 (18.9)
13 (35.1)
17 (45.9)
Send post-interview emails to URiMs only
175
34 (19.4)
1 (2.9)
16 (47.1)
17 (50.0)
Other
11
9 (81.8)
3 (33.3)
6 (66.7)
0 (–)
ERAS = Electronic Residency Application System; URiM = underrepresented in medicine applicants.
Note: Results are presented as n (%) across the rows. Denominator for assessment of effectiveness is the number of respondents who reported using the strategy.
Cronbach's alpha (α) for items reported: 0.81; average interitem covariance: 0.008.
Perspectives on the effectiveness of each of these strategies varied; no single strategy dominated in terms of being perceived as “very effective.” However, the strategies reported by the largest percentages of respondents as “very effective” or “somewhat effective” were websites featuring diversity, mentioning diversity explicitly on interview day, featuring URiM residents and faculty on interview day, using race/ethnicity ERAS data, matching URiM applicants with URiM interviewees, and recruitment at URiM association events. In contrast, strategies that involved directly contacting URiM applicants were not only the strategies reported to be least used, but were also perceived as least effective. Such strategies included making individual phone calls after interview day to URiM applicants only, special meetings with program leadership for URiM applicants prior to and on interview day, and holding separate events or second visits for URiM applicants.
Program directors reported several barriers to recruitment, summarized in Table 3. The most cited barriers were concerns about applicant interest in the geographic region of the residency program, diversity of the applicant pool, and qualifications of the applicant pool. In addition to the structured responses, some respondents who reported an item for “other” referenced concerns about the institution itself being a barrier to recruitment (eg, being part of the military, being a new institution, and losing out to other institutions).
Table 3Barriers to Recruitment of URiM Applicants to Residency
In response to a question about whether the ACGME requirements would impact their actions, 25% (73/293) reported “yes” that they had made a change to their programs, and 52/293 (18%) were unsure; 36/60 (60%) reported that they were planning to implement a formal tracking system for recruitment and retention, 33/60 (55%) reported that they would implement committees to work on recruitment and retention, 6/60 (10%) reported they were seeking more funding. For 167 of 168 respondents who reported that they had not made a change to their recruitment efforts, 90 (54%) stated that their current strategy met requirements, 34 (20%) reported that their strategy exceeded requirements, and 19 (11%) were waiting on an institutional plan. For reports of “other,” 6 referenced IMGs and 3 referenced military programs.
When asked what respondents wished their institution would focus on to improve URiM recruitment, the responses clustered into 6 major themes (Table 4). Of 155 respondents to this question, 44 (28%) reported a need to support faculty diversity, 29 (19%) emphasized that a culture shift was necessary, 34 (22%) desired more support through resources and education, 21 (14%) suggested focusing on the low numbers in the pipeline, 16 (10.3%) felt they were comfortable with their level of diversity, and 11 (7%) felt IMGs added diversity but were unaccounted for in traditional URiM metrics.
Table 4Themes from Self-Reported Institutional Efforts to Improve URiM Recruitment
Open-ended responses to the question, “What is one area that you wish your institution would focus on to improve recruitment of URiM trainees?”
Themes
N = 155, n (%)
Representative quotes
Culture change viewed as imperative to recruitment
29 (18.7)
•
“Mandate that it is a priority at each level of leadership. We are grinding our wheels with a grassroots program.”
•
“Support the programs in changing the criteria for recruitment.”
•
“Be more progressive in the ways in which they work to improve recruitment of URiM trainees.”
The value of international medical graduates in augmenting diversity
11 (7.1)
•
“[We] have been very ethnically diverse. Now whether that meets the definition of URiM trainees is unclear.”
•
“[We] are a 100% IMG program and have very few Americans.”
•
“I am unsure how to attract any US graduates, including URiM graduates to our program when the national trend and the local competition is significant.”
The need to emphasize faculty diversity and support
44 (28.4)
•
“I wish that they would give protected time to URiM Faculty to mentor URiM residents. Our faculty are SO STRETCHED [sic] because of increasing clinical responsibilities that it is completely unrealistic to think that they will be able to commit to meaningful engagement to the residents in their free time. We have to deal HEAD ON [sic] with the minority tax issue.”
•
“We need more representation in our faculty.”
The desire to increase support through resources and education
34 (21.9)
•
“Providing additional resources in terms of greater housing accommodations, loan forgiveness counseling/plans, and more faculty of color so the trainees can see themselves in the institution.”
•
“Allowing resources for separate recruitment efforts and providing time/opportunity/protected. Funding for diverse faculty to step away from clinical duties to attend recruitment duties.”
The perception of having already accomplished the mission of diversity
16 (10.3)
•
“Our program is very mission driven and this reflects greatly on our ability to attract diverse applicants.”
•
“Our hospital serves a very diverse community. This is reflected in our recruitment process and the diversity of our residents.”
Concerns about the insufficient number of URiM applicants and limitations of pipeline programs
21 (13.5)
•
“Stronger outreach to regional medical schools regarding opportunities for URiM in our health care organization/system.”
•
“Recruitment of local minorities into medical school to have better chance of matching them in our residency program.”
•
“Need greater diversity of applicant pool with qualified applicants.”
IMG = international medical graduate; URiM = underrepresented in medicine applicants.
Open-ended responses to the question, “What is one area that you wish your institution would focus on to improve recruitment of URiM trainees?”
We conducted a national survey of internal medicine residency program directors to identify strategies used to recruit historically URiM applicants. We found that intentional efforts on interview day were frequent, such as verbalizing commitment to diversity and inclusion and featuring URiM residents. More resource-intensive approaches were uncommon, such as separate events for URiMs, post-interview communications, and second visits, and had varying degrees of perceived effectiveness. Barriers identified in the process of URiM recruitment centered on concerns about the applicant pool. Collectively, these perspectives represent a toolbox of initiatives for residency programs seeking to increase the diversity of their programs.
It is not surprising that the most reported recruitment strategies occurred on interview day because it is the predominant opportunity for programs to engage with applicants. These strategies were explicit in nature—websites, the presence of URiM residents, verbally highlighting diversity—and have 2 sides. On one hand, programs may have wanted to broadcast their commitment to diversity; on the other, the emphasis on visual display may have reflected surface-level attitudes toward diversity that do not necessarily demonstrate value or commitment.
We found that “typical” barriers for initiatives, namely, funding and external support, were not common issues among our respondents. Rather, respondents reported the applicant pool and applicant interest in geography to be the largest perceived barriers. This perspective requires corroboration from applicants. Medical schools, although separate from the residency programs, were reported as the greatest source of support from diversity offices, although it was not highly utilized, and it underscores the need to work with other faculty and administrative leaders to further pipelines and decrease siloed efforts.
Our work demonstrated an unexpected but persistent set of themes related to IMGs, centered on whether they could be considered URiMs. Some program directors felt that IMGs clearly brought diversity to programs, whereas others articulated that having a large proportion of IMGs reflected an inability to attract URiMs. Of note, the ACGME definition of “underrepresented” has not been prescriptive. Overall, there was a strong signal from a subset of respondents that IMGs had the potential to infuse ethnic, cultural, and linguistic diversity into programs, a view that may not be universally held.
Notably, a recent qualitative study of 20 internal medicine program directors revealed a lack of familiarity with the ACGME diversity standards, concerns about the absence of national guidance for recruitment, and apprehensions about match violation with post-interview contact.
Its findings also focused more on barriers to recruitment rather than effective strategies, which differentiates our work. This work also builds on the prior survey of obstetrics-gynecology program directors, representing a larger collection of program director efforts to improve URiM recruitment. Our pair of studies suggest that using ERAS race/ethnicity and promoting diversity on interview day are key to building a diverse cohort, similar to an initiative in family medicine.
may be in the eye of the beholder, and our results accentuate the lack of agreement on this point.
Limitations to this work include methodological constraints inherent to surveys, which include measurement error, possible non-response bias, and the self-reporting of perceptions. The survey was deployed during application season; however, respondents may have experienced recall bias about the breadth of strategies used to recruit URiM applicants. Further, respondents presented subjective views about the effectiveness of particular strategies, which may have been influenced by confirmation bias in that they had personal stakes in affirming the value of efforts they adopted. Similarly, social acceptance bias may have affected the magnitude of their responses, particularly questions about how much they emphasized race and minority recruitment. Additionally, there were varying degrees of item non-response for our survey section questions; the subset most committed to diversity may have chosen to answer our questions about URiM recruitment and thus, our findings may not apply to less invested programs. That said, the nonresponders may not have done so due to their relative newness, rather than lack of interest.
This contribution to a national survey characterized recruitment strategies to diversify the physician workforce. A subsequent research agenda should include a systematic linkage of recruitment strategies to objective metrics, with the potential for a resource-effectiveness evaluation. In addition to crowdsourcing local solutions to a national problem, the findings draw attention to opportunities for the specialty as whole to improve the demographic disparity between patients and physicians.
Acknowledgment
We are grateful to the Association of Program Directors in Internal Medicine for their annual survey mechanism, without which the project would not have been possible.
References
Saha S
Guiton G
Wimmers PF
Wilkerson L
Student body racial and ethnic composition and diversity-related outcomes in US medical schools.
American Medical Association. Fellowship and residency electronic interactive database access system online. Available at: https://freida.ama-assn.org/. Accessed October 3, 2021.
Bivariate (Adjusted Wald [Pearson] test of association with 1 degree of freedom) used for categorical variables: alpha = 0.05.
00150-4&id=gr1.jpg){kind=link}