Advertisement
The American Journal of Medicine
Advanced SearchSave search
AAIM Perspectives| Volume 135, ISSUE 2, P258-263.e1, February 01, 2022

Virtual Recruitment: Experiences and Perspectives of Internal Medicine Program Directors

Published:November 06, 2021DOI:https://doi.org/10.1016/j.amjmed.2021.10.017
Virtual Recruitment: Experiences and Perspectives of Internal Medicine Program Directors
Previous ArticleAssociation of Social Gaming with Well-Being (Escape COVID-19): A Sentiment Analysis
Next ArticleHonoring Medicine's Fathers

      Keywords

      Perspectives Viewpoints
      • •
        Virtual recruitment is new to nearly all internal medicine residency programs.
      • •
        The majority of internal medicine program directors were equally or more satisfied with their 2020-2021 incoming intern class compared with the prior 3 intern classes.
      • •
        Cost savings for residency programs are substantial for virtual interviews, compared to in-person interviews.
      • •
        Internal medicine program directors express varying opinions about the future of virtual and in-person recruitment.

      Introduction

      Due to the coronavirus disease 2019 (COVID-19) pandemic, internal medicine residency recruitment in fall 2020 was exclusively conducted virtually.

      Association of American Medical Colleges (AAMC). Final Report and Recommendations for Medical Education Institutions of LCME-Accredited, U.S. Osteopathic, and Non-U.S. Medical School Applicants. Available at:https://www.aamc.org/system/files/2020-05/covid19_Final_Recommendations_05112020.pdf. Accessed August 2, 2021.

      This change represented a major shift in recruitment strategies and operations for US training programs, with limited data on best practices for virtual interviews.

      Huppert LA, Hsiao EC, Cho KC, et al. Virtual interviews at graduate medical education training programs: determining evidence-based best practices [e-pub ahead of print]. Acad Med. doi:10.1097/acm.0000000000003868. Accessed September 28, 2021.

      ,
      • Wolff M
      • Burrows H
      Planning for virtual interviews: residency recruitment during a pandemic.
      Acad Pediatr. 2021; 21: 24-31https://doi.org/10.1016/j.acap.2020.10.006
      Prior to COVID-19, applicants routinely traveled for in-person interviews with program faculty, met with program leadership and current residents, and visited the location and health care facilities. The in-person interview process provided applicants the benefit of learning about the culture and multiple aspects of the program and its location and had a major influence on how applicants ranked programs.

      National Resident Matching Program. National Resident Matching Program, Data Release and Research Committee: Results of the 2019 NRMP Applicant Survey by Preferred Specialty and Applicant Type. Available at:https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2019/06/Applicant-Survey-Report-2019.pdf. Accessed August 2, 2021.

      In 2020, conditions associated with the pandemic demanded a change to a virtual interview process with preparation occurring in a compressed timeline that coincided with unpredictable COVID-19 surges.
      Intern recruitment is a core responsibility for internal medicine program directors

      Accreditation Council for Graduate Medical Education (ACGME. Common Program Requirements (Residency). Available at:https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResidency2020.pdf. Accessed August 2, 2021.

      and a sizable undertaking with significant investments of time and resources by residency programs and applicants. Prepandemic, internal medicine programs spent a median of $14,162 per filled intern position with an overall median cost of recruitment per program at $148,000.
      • Brummond A
      • Sefcik S
      • Halvorsen AJ
      • et al.
      Resident recruitment costs: a national survey of internal medicine program directors.
      Am J Med. 2013; 126: 646-653https://doi.org/10.1016/j.amjmed.2013.03.018
      Added to the costs, students, medical schools, and program directors have grappled with escalating numbers of residency applications for several years.
      • Antono B
      • Willis J
      • Phillips Jr, RL
      • Bazemore A
      • Westfall JM
      The price of fear: an ethical dilemma underscored in a virtual residency interview season.
      J Grad Med Educ. 2021; 13: 316-320https://doi.org/10.4300/jgme-d-20-01411.1
      In 2020, the mean number of Electronic Residency Application Service (ERAS) applications submitted for each applicant to internal medicine programs had risen to a high of 70.7 from 59.3 in 2016,

      Association of American Medical Colleges (AAMC). ERAS Statistics 2016-2020- Internal Medicine (Categorical). Available at: https://www.aamc.org/data-reports/interactive-data/eras-statistics-data. Accessed August 16, 2021.

      representing approximately $1500 in direct application costs per applicant,

      Association of American Medical Colleges (AAMC). Fees for ERAS Residency Applications. Available at:https://students-residents.aamc.org/applying-residencies-eras/fees-eras-residency-applications. Accessed September 13, 2021.

      which does not include travel. In response to this “application inflation,” some program directors have increased the number of invitations extended and interviews conducted and have raised their screening standards for interview offers.
      • Angus SV
      • Williams CM
      • Stewart EA
      • Sweet M
      • Kisielewski M
      • Willett LL
      Internal medicine residency program directors' screening practices and perceptions about recruitment challenges.
      Acad Med. 2020; 95: 582-589https://doi.org/10.1097/ACM.0000000000003086
      ,
      • Sweet ML
      • Williams CM
      • Stewart E
      • et al.
      Internal medicine residency program responses to the increase of residency applications: differences by program type and characteristics.
      J Grad Med Educ. 2019; 11: 698-703https://doi.org/10.4300/JGME-D-19-00194.1
      Virtual recruitment imposed another dimension of uncertainty in an already cost- and labor-intensive process.
      Limited data are available about the effects of the wholesale shift to virtual recruitment and how it might affect future recruitment. To better understand this, we conducted a national survey of internal medicine program directors about their experiences and program outcomes in the first completely virtual recruitment season. Further, we solicited program director perspectives about application inflation as well as guiding principles for future recruitment seasons.

      Methods

      The Association of Program Directors in Internal Medicine (APDIM), a professional organization that represents internal medicine residency program directors, core faculty, and program administrators, is a charter organization of the Alliance for Academic Internal Medicine (AAIM). The APDIM Survey Committee regularly surveys internal medicine program directors on a variety of critical topics in graduate medical education (GME).
      • Marshall AL
      • Halvorsen AJ
      • Kearns L
      • Reich LM
      • West CP
      • Oxentenko AS
      Disparities in salary and work-life integration in internal medicine program directors are associated with gender and partner employment status.
      Am J Med. 2021; 134: 285-291.e1https://doi.org/10.1016/j.amjmed.2020.10.004

      Survey Instrument

      In fall 2020, the APDIM Survey Committee recognized the need to conduct a survey of internal medicine program directors regarding the residency virtual interview and recruitment process. The tentative subject matter for the survey was curated from exchanges via the APDIM Discussion Forum (an email listserv including more than 4000 GME faculty and administrators) and the final topics were identified via committee consensus. The initial survey sections were drafted by a subset of committee members, and the complete survey was reviewed and revised by the entire committee. The instrument was programmed in the Qualtrics Surveys

      Qualtrics. Qualtrics Software. Version XM. Provo, UT: Qualtrics; 2020.

      platform by AAIM staff and was pilot-tested by the committee as well as 6 GME experts outside of the committee. The instrument consisted of 25 questions with conditional (skip or display) logic and included multiple-choice-select-one, 5-point Likert scale, numeric entry, and open-text response questions.
      In March 2021, the survey was distributed via a personalized email invitation to 436 internal medicine residency program directors, representing 82% (436 of 532) of US internal medicine residency programs with Accreditation Council for Graduate Medical Education (ACGME) accreditation prior to July 1, 2019 (ie, at least 1 year prior to the academic year to which the survey applied). Four email reminders to nonrespondents were sent, and the survey closed in May 2021. The study (21-AAIM-117) was deemed exempt by Pearl Institutional Review Board (US DHHS OHRP #IRB00007772) under FDA 21 CFR 56.104 and 45CFR46.104(b)(2).

      Data Analysis

      Data analysis was conducted in Stata 16.1 SE
      Stata Corp
      Stata Statistical Software. Version Release 16.
      StataCorp LLC, College Station, TX2019
      by AAIM staff. Before deidentifying the final responses for analysis, the study data set was appended with data from external sources that explained the most survey population variance, including US Census Bureau geographic region.

      US Census Bureau. Census Regions and Divisions of the United States. Available at: https://www2.census.gov/geo/pdfs/maps-data/maps/reference/us_regdiv.pdf. Accessed December 1, 2020.

      Residency program characteristics such as number of approved positions were obtained from the Accreditation Council for Graduate Medical Education Accreditation Database System (Public).

      Accreditation Council for Graduate Medical Education (ACGME). Accreditation Database System Online (Public). Available at: https://apps.acgme.org/ads/Public/Programs/Search. Accessed June 1, 2020.

      Program type and other program characteristics were obtained from the American Medical Association Fellowship and Residency Electronic Interactive Database Access System Online.

      American Medical Association (AMA). Fellowship and Residency Electronic Interactive Database Access System (FREIDA) Online. Available at: https://freida.ama-assn.org/search/list?spec=42771. Accessed September 1, 2020.

      American Board of Internal Medicine (ABIM) rolling 3-year residency pass rates were provided by the board.

      American Board of Internal Medicine (ABIM).Residency Program Pass Rates [2008-2019]. Available at:https://www.abim.org/Media/lhgmdidp/residency-program-pass-rates.pdf. Accessed June 2020.

      Summary statistics included frequencies and percentages for categorical variables and measures of central tendency (eg, mean, median, standard deviation, interquartile range) for continuous variables. To assess the statistical representativeness of the responses, essential program characteristics from the data sources described previously were used to compare respondents and their programs to the survey-eligible population. We tested for goodness-of-fit or associations between categorical variables using the Adjusted Wald (Pearson) χ2 test (1 degree of freedom); a design-based Pearson χ2 (2 degrees of freedom) was used to assess goodness-of-fit when both categorical variables exceeded 2 categories. Due to unequal variances, we used the Welch t-test to compare the means of continuous variables against dichotomous variables. Statistical significance was designated with an alpha level of P ≤ .05. Due to conditional logic or item nonresponse, some denominators do not equal the total number of survey respondents.

      Results

      The response rate was 61.9% (270 of 436 responses). There was no statistical association between respondents and nonrespondents based on essential characteristics that defined the complete survey population (Table 1). Although there was slight over-representation of university-based programs (39.3% among respondents; 31.9% for the population), that difference was not statistically significant (P = .053).
      Table 1Essential Program Characteristics of Respondents and Nonrespondents: 2021 APDIM Spring Survey of Internal Medicine Residency Program Directors
      Respondents (n = 270)Nonrespondents (n = 166)Total (n = 436)
      CharacteristicNo. (Column %)No. (Column %)No. (Column %)P Value
      Bivariate (Adjusted Wald [Pearson]) test of association with 1 degree of freedom).
      Program Type (AMA-FREIDA)
       University-based106 (39.3)33 (19.9)139 (31.9).053
       Community-based37 (13.7)43 (25.9)80 (18.4).109
       Community-based, university-affiliated122 (45.2)89 (53.6)211 (48.4).324
       Military-based5 (1.9)1 (0.6)6 (1.4).349
      Census Region (US Census Bureau)
      Excludes programs from 2 US territories, due to small cell sizes/data confidentiality.
       Northeast83 (30.7)51 (30.7)134 (30.7).998
       Midwest60 (22.2)38 (22.9)98 (22.5).922
       West41 (15.2)27 (16.3)68 (15.6).769
       South86 (31.9)50 (30.1)136 (31.2).842
      VA Affiliation: Yes (ACGME)107 (39.6)47 (28.3)154 (35.3).096
      Accreditation Status (ACGME)
       Continued or continued with warning260 (96.3)152 (91.6)412 (94.5).099
       Initial or initial with warning10 (3.7)14 (8.4)24 (5.5)
      Mean (SD)Mean (SD)Mean (SD)P Value
      Welch t-test with unequal variances.
      Program size: No. ACGME approved positions (Q3-Q1: 38-102; 30-69; 36-86)
      Interquartile range test (Welch t-test).
      		56.2 (18.1)53.1 (21.2)55.0 (20.0).177
      ABIM pass rate: 2017-2019 (%); n = 248, n = 141, n = 38992.4 (6.3)89.4 (10.8)91.3 (8.2).999
      Program director tenure as of March 2021 (years: ACGME)7.0 (6.1)6.1 (5.7)6.7 (5.9).101
      Program accreditation year (ACGME) (Q3-Q1: 1956-1979; 1960-2015; 1957-2001)
      Interquartile range test (Welch t-test).
      		1969.6 (9.8)1971.4 (11.6)1969.9 (10.5).120
      Average USMLE Step 1 Score (FREIDA); n = 214, n = 131, n = 345212.5 (11.5)213.9 (12.5)213.0 (11.9).252
      ABIM = American Board of Internal Medicine; ACGME = Accreditation Council for Graduate Medical Education; AMA-FREIDA = American Medical Association Residency and Fellowship Database; APDIM = Association of Program Directors in Internal Medicine; SD= standard deviation; USMLE = US Medical Licensing Examination; VA = Veterans Affairs.
      low asterisk Bivariate (Adjusted Wald [Pearson]) test of association with 1 degree of freedom).
      † Excludes programs from 2 US territories, due to small cell sizes/data confidentiality.
      ‡ Welch t-test with unequal variances.
      § Interquartile range test (Welch t-test).
      Ninety percent of programs (243 of 270) reported that virtual interviews were not employed routinely prior to the 2020-2021 recruitment season. Compared to the 2019-2020 recruitment season, a majority of program directors reported receiving more applications from US medical school students (59.1%, 159/269) and offering more interviews (61.5%, 166/270). The median percentage increase in categorical position interviews offered was 20% (mean: 22, standard deviation: 16.3). Sixty-five percent of respondents (174 of 270) reported that fewer applicants declined or canceled interviews compared to the prior year.
      Applicant favorability varied, defined by asking where on the rank list their last matched applicant was compared to their program's average match over the past 3 years. Of program directors who could answer this question, 32.8% (87/265) reported that the last matched position was more favorable, 37.7% (100 of 265) reported it was about the same, and 29.5% (78 of 265) reported it was less favorable. Reported favorability of the last matched position varied by geographic region (Table 2) but did not vary by median program size, type, or program director's tenure (median years in position) (data not shown). For program directors who reported less favorable matches, most (85.9%, 67 of 78) attributed that outcome to a “moderate” or “great” extent to virtual recruitment. For program directors who reported more favorable matches, a minority (27.6%, 24 of 87) attributed that outcome to a “moderate” or “great” extent to virtual recruitment. Overall, 81.2% of program directors (216 of 266) reported that they were similarly or more satisfied with their incoming intern class compared with the past 3 years; the responses of which did not vary by program type (Appendix A, available online).
      Table 2Regional Differences in the Location of Last Matched Applicant on the Rank List (2020-2021 Recruitment Cycle) Compared to the Previous 3 Years
      Residency Program US Census RegionMuch More Favorable/More Favorable (n = 87)About the Same (n = 100)Less Favorable/Much Less Favorable (n = 78)Total (n = 265)P Value
      Design-based Pearson χ2 goodness-of-fit test with 2 degrees of freedom.
      Midwest18 (20.7)21 (21.0)21 (26.9)60 (22.6).552
      Northeast26 (29.9)42 (42.0)12 (15.4)80 (30.2).035
      South36 (41.4)22 (22.0)27 (34.6)85 (32.1).035
      West7 (8.1)15 (15.0)18 (23.1)40 (15.1).019
      Note: For 265 of 270 respondents; an additional 5 respondents reported “Unsure or Not applicable.”
      low asterisk Design-based Pearson χ2 goodness-of-fit test with 2 degrees of freedom.
      Virtual recruitment affected what program directors were able to convey about their programs and what they learned about applicants. Most program directors indicated that virtual recruitment reduced their ability to showcase their program's facilities (89.9%, 240/267), city (87.7%, 235/268), and esprit de corps (82%, 219/267). Program directors (71.3%, 189/265) reported a decreased ability to determine which applicants had a sincere interest in their program. In contrast, the majority of program directors reported that the ability to highlight their program's defining features was unchanged (50.3%, 135 of 268) or had increased (14.6%, 39 of 268).
      Excluding program directors who responded “unsure or not applicable,” almost all (94%, 250 of 266) reported a somewhat or substantial decrease in recruitment costs, and 40.9% of program directors (108 of 264) reported increased ability to get faculty to conduct interviews. Slightly less than half of program directors (46.6%, 125 of 270) reported investing more of their time for interview days.
      Although a majority of program directors (74.4%, 200 of 269) reported to favor or strongly favor a guiding set of principles on whether internal medicine recruitment should be conducted virtually, in-person, or in a combined fashion, there was not broad consensus about what those principles should be (Appendix B, available online). A minority of program directors endorsed a fully virtual (5.6%, 15 of 269) or a fully in-person (7.4%, 20 of 269) recruitment process. Twenty-one percent (55 of 269) of program directors expressed that programs should make their own determinations and the remainder (28.6%, 164 of 269) preferred a hybrid option. Excluding respondents who reported “unsure,” more than half reported that if unrestricted travel and hospital visitors are allowed for the next recruitment season, they would retain virtual interviews with faculty (59.7%, 151 of 253), videos featuring the program or city (88.5%, 224 of 253), and a fully virtual interview option (61.3%, 155 of 253).
      Program directors reported that virtual recruitment contributed to greater (79.5%, 214 of 269) application inflation. Two-thirds (68%, 183 of 269) agreed with the statement that internal medicine residency programs should pursue voluntary preference signaling, whereas 11.2% (30 of 269) reported “neutral,” 16% (43 of 269) disagreed, and 4.8% reported “unsure.”

      Discussion

      This national survey of internal medicine program directors demonstrated that although virtual recruitment was new to most residency programs, the vast majority (more than 80%) of respondents indicated satisfaction with their incoming class of interns. Respondents reported more applicants to their programs (compared to the previous recruitment season), an increased number of interviews conducted, fewer canceled or declined interviews, and the perspective that virtual interviewing contributed to application inflation. Among program directors who reported less favorable match outcomes, virtual recruitment was perceived to be a contributing factor and, conversely, less so for program directors who reported more favorable match outcomes. Benefits of virtual recruitment were several and included lower costs for nearly all programs and increased ease of faculty participation in recruitment. However, almost half of responding program directors invested more of their time on the interview day. Program directors indicated that it was more challenging to communicate key aspects of their program such as esprit de corps. Although program directors did not express agreement about what principles should shape recruitment in the future, many planned to retain some aspects of virtual recruitment.
      Our results are in keeping with the results of a 2021 National Residency Matching Program (NRMP) cross-specialty survey

      National Resident Matching Program. 2021 Applicant and Program Director Survey Findings: Impact of the Virtual Experience on the Transition to Residency. Available at:https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2021/05/Research-Brief-Virtual-Experience-2021-FINAL.pdf. Accessed August 17, 2021.

      and preliminary 2021 ERAS data on internal medicine recruitment.

      Association of American Medical Colleges (AAMC). Supplemental ERAS application (for the ERAS 2022 cycle). Available at: https://students-residents.aamc.org/applying-residencies-eras/supplementalerasapplication. Accessed September 7, 2021.

      Broadly, approximately one-third of program directors indicated that their programs received more applications, invited more applicants, and had fewer cancellations during the 2020-2021 recruitment cycle. For internal medicine residency and many internal medicine subspecialty fellowships, the number of applicants further increased in 2021.
      • Huppert LA
      • Santhosh L
      • Babik JM
      Trends in US internal medicine residency and fellowship applications during the COVID-19 pandemic vs previous years.
      JAMA Netw Open. 2021; 4e218199https://doi.org/10.1001/jamanetworkopen.2021.8199
      Internal medicine applications rose by 6% and the mean number of applications submitted by an internal medicine applicant increased by 11% compared to the prior year.
      • Huppert LA
      • Santhosh L
      • Babik JM
      Trends in US internal medicine residency and fellowship applications during the COVID-19 pandemic vs previous years.
      JAMA Netw Open. 2021; 4e218199https://doi.org/10.1001/jamanetworkopen.2021.8199
      Similar to internal medicine program directors perspectives in our study, about three-quarters of all residency applicants in a 2021 National Residency Matching Program survey found it challenging to develop an understanding of program culture and degree of fit.

      National Resident Matching Program. 2021 Applicant and Program Director Survey Findings: Impact of the Virtual Experience on the Transition to Residency. Available at:https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2021/05/Research-Brief-Virtual-Experience-2021-FINAL.pdf. Accessed August 17, 2021.

      It remains to be seen what effect this has on resident satisfaction over time. Despite these issues, internal medicine program directors reported satisfaction with their match results. Program directors of colon and rectal surgery programs noted a high degree of satisfaction with match outcomes.

      D'Angelo JD, D'Angelo AD, Mathis KL, Dozois EJ, Kelley SR. Program director opinions of virtual interviews: whatever makes my partners happy [e-pub ahead of print]. J Surg Educ. doi:10.1016/j.jsurg.2021.04.008. Accessed August 17, 2021.

      Otherwise, few studies have reported program directors perceptions of their match results.
      It is interesting that program directors who reported less favorable matches attributed it to virtual recruitment in contrast to those who reported more favorable matches and perceived that virtual recruitment played less of a role. Attributing performance of the virtually recruited intern classes solely to virtual recruitment (favorably or unfavorably) overlooks many confounding factors and likely focuses on an obvious but not comprehensive assessment of trainees.
      Virtual recruitment likely benefits some programs and not others. It may also explain the variety of opinions reported by program directors about best practices moving forward: For program directors who believed there should be a national policy, only 5.6% expressed a preference for a purely virtual process, suggesting that getting to know applicants and “selling” the culture, facility, and city is important.
      Cost savings for applicants and programs are one of the major advantages of virtual recruitment. In this study, virtual interviewing was not associated with a decrease in program directors time spent on interview days, which may be due to the newness of virtual recruitment for 90% of programs. Additionally, the environmental footprint of recruitment was substantially reduced
      • Donahue LM
      • Morgan HK
      • Peterson WJ
      • Williams JA
      The carbon footprint of residency interview travel.
      J Grad Med Educ. 2021; 13: 89-94https://doi.org/10.4300/jgme-d-20-00418.1
      as carbon emissions are estimated to be 51,665 metric tons of carbon dioxide per year for residency interviews across the United States.
      A majority of internal medicine responding program directors favored implementation of some form of signaling preference pathway in response to ongoing application inflation, a problem seemingly exacerbated by virtual recruitment. A signaling preference pathway—proposed as a solution to application inflation—was piloted by otolaryngology in 2020-2021 through a voluntary, free preference-signaling program.

      Society of University Otolaryngologists. Otolaryngology Preference Signaling. Available at: https://opdo-hns.org/mpage/signaling. Accessed August 18, 2021.

      A previous study demonstrated that two-thirds of internal medicine program directors were interested in having a mechanism for applicants to indicate a high level of interest.
      • Angus SV
      • Williams CM
      • Stewart EA
      • Sweet M
      • Kisielewski M
      • Willett LL
      Internal medicine residency program directors' screening practices and perceptions about recruitment challenges.
      Acad Med. 2020; 95: 582-589https://doi.org/10.1097/ACM.0000000000003086
      For the 2022 recruitment season, the Association of American Medical Colleges (AAMC) is piloting a voluntary supplemental ERAS application in surgery, dermatology, and internal medicine

      Association of American Medical Colleges (AAMC). Supplemental ERAS application (for the ERAS 2022 cycle). Available at: https://students-residents.aamc.org/applying-residencies-eras/supplementalerasapplication. Accessed September 7, 2021.

      that will include preference signaling for up to 5 programs.
      To our knowledge, this study is the first to describe internal medicine program director experiences with virtual recruitment and to study their opinions about program outcomes and future plans. However, our study has limitations. The 2020-2021 recruitment season was the first conducted virtually; practices and program directors opinions may change substantially over time. Additionally, not only were program directors modifying their recruitment practices, but they were also responding to a pandemic in which their trainees were on the front lines of care. This pressure may have significantly influenced their experience with and the implementation of virtual recruitment. Although the survey response rate was generally representative of the underlying population, possible nonresponse bias does not allow the results to be completely reflective of the perspectives and experiences of all internal medicine program directors at the time of the survey.
      Now that there is more planning time and given that a hybrid recruitment approach may be possible in the future, further studies are needed as program directors refine their approach to recruitment and incorporate virtual and in-person elements. Further, virtual recruitment's impact on diversity and equity in internal medicine residency selection is not yet known.
      • Hammoud MM
      • Standiford TC
      • Carmody JB
      The 2020-2021 residency application cycle: Lessons learned and lingering problems.
      JAMA. 2021; 325: 2249-2250https://doi.org/10.1001/jama.2021.5708
      , 
      • Nwora C
      • Allred DB
      • Verduzco-Gutierrez M
      Mitigating bias in virtual interviews for applicants who are underrepresented in medicine.
      J Natl Med Assoc. 2021; 113: 74-76https://doi.org/10.1016/j.jnma.2020.07.011
      , 

      Heitkamp NM, Snyder AN, Ramu A, et al. Lessons learned: applicant equity and the 2020-2021 virtual interview season. Acad Radiol. doi:10.1016/j.acra.2021.08.005. Accessed September 29, 2021.

      In summary, virtual recruitment offers significant cost savings for residency programs and applicants as well as reduced environmental impact, and most program directors reported satisfaction with the outcome. However, it may be more difficult with virtual recruitment to convey all program traits and to assess the extent of applicant interest. Future internal medicine recruitment will be shaped by the experiences of the 2020-2021 recruitment season. Like some other specialties,

      Hamade N, Bhavsar-Burke I, Jansson-Knodell C, et al. Virtual gastroenterology fellowship recruitment during COVID-19 and its implications for the future [e-pub ahead of print]. Dig Dis Sci. doi:10.1007/s10620-021-07014-1. Accessed August 17, 2021.

      , 
      • Hill MV
      • Ross EA
      • Crawford D
      • et al.
      Program and candidate experience with virtual interviews for the 2020 Complex General Surgical Oncology interview season during the COVID pandemic.
      Am J Surg. 2021; 222: 99-103https://doi.org/10.1016/j.amjsurg.2020.11.007
      , 
      • Jimenez AE
      • Khalafallah AM
      • Romano RM
      • et al.
      Perceptions of the virtual neurosurgery application cycle during the coronavirus disease 2019 (COVID-19) pandemic: a program director survey.
      World Neurosurg. 2021; 154: e590-e604https://doi.org/10.1016/j.wneu.2021.07.078
      , 

      Rajesh A, Asaad M, Elmorsi R, Ferry AM, Maricevich RS. The virtual interview experience for MATCH 2021: a pilot survey of general surgery residency program directors [e-pub ahead of print]. Am Surg. doi:10.1177/00031348211038555. Accessed September 30, 2021.

      , 
      • Rhoades JS
      • Ramsey PS
      • Metz TD
      • Lewkowitz AK
      Maternal-fetal medicine program director experience of exclusive virtual interviewing during the coronavirus disease 2019 pandemic.
      Am J Obstet Gynecol MFM. 2021; 3100344https://doi.org/10.1016/j.ajogmf.2021.100344
      , 

      Geary AD, Wang TS, Lindeman B, et al. Perspectives on virtual interviews-a follow-up study of the Comprehensive Endocrine Surgery Fellowship interview process [e-pub ahead of print]. Surgery. doi:10.1016/j.surg.2021.03.069. Accessed September 30, 2021.

      many internal medicine program directors plan to incorporate virtual interviews into future recruitment cycles. This shift suggests that a hybrid approach may become the norm even after the COVID-19 pandemic ends. Application inflation remains a major challenge for both internal medicine programs and applicants and ostensibly was exacerbated by virtual recruitment. The upcoming ERAS supplemental application pilot is an important effort to address this challenging issue, although multiple strategies are likely needed.

      Acknowledgments

      We would like to thank the members of the Alliance for Academic Internal Medicine–Association of Program Directors in Internal Medicine Survey and Scholarship Committee for its support in reviewing and revising the survey instrument that informed this research, as well as the residency program directors who completed the survey.

      Appendix A Reported Satisfaction Level with the Incoming Intern Class (2020-2021 Recruitment Cycle) by Residency Program Type

      Tabled 1
      University-based (n = 105)Community-based (n = 37)Community-based, university-affiliated (n = 120)Total (n = 262)P Value
      Design-based Pearson χ2 goodness-of-fit test with 2 degrees of freedom
      Much more/somewhat more satisfied39 (37.1)14 (37.8)44 (36.7)97 (37.0).989
      About the same49 (46.7)12 (32.4)54 (45.0)115 (43.9).229
      Somewhat more/much more dissatisfied17 (16.2)11 (29.7)22 (18.3)50 (19.1).104
      For 262 of 270 respondents (4 nonrespondents). An additional 4 respondents reported “Unsure/Unable to answer.” Excludes 4 military-based programs due to small cell sizes/data confidentiality.
      low asterisk Design-based Pearson χ2 goodness-of-fit test with 2 degrees of freedom

      Appendix B Guiding Set of Principles Endorsed by Residency Program Directors

      Tabled 1
      n = 269
      Recruitment should be only in person (if safe due to COVID-19)20 (7.4)
      Virtual option should be offered even if in-person is the norm again77 (28.6)
      Recruitment should remain virtual with option for subsequent onsite visit87 (32.3)
      Recruitment should be only virtual15 (5.6)
      Each program should be allowed to make its own determination55 (20.5)
      Unsure15 (5.6)
      Notes: For 269 of 270 respondents; 1 nonrespondent.
      The survey question asked “If there is general agreement in favor of a guiding set of principles, which of the following would you endorse?”

      References

      1. Association of American Medical Colleges (AAMC). Final Report and Recommendations for Medical Education Institutions of LCME-Accredited, U.S. Osteopathic, and Non-U.S. Medical School Applicants. Available at:https://www.aamc.org/system/files/2020-05/covid19_Final_Recommendations_05112020.pdf. Accessed August 2, 2021.

        View in Article

      2. Huppert LA, Hsiao EC, Cho KC, et al. Virtual interviews at graduate medical education training programs: determining evidence-based best practices [e-pub ahead of print]. Acad Med. doi:10.1097/acm.0000000000003868. Accessed September 28, 2021.

        View in Article
        • Wolff M
        • Burrows H
        Planning for virtual interviews: residency recruitment during a pandemic.
        Acad Pediatr. 2021; 21: 24-31https://doi.org/10.1016/j.acap.2020.10.006
        View in Article

      4. National Resident Matching Program. National Resident Matching Program, Data Release and Research Committee: Results of the 2019 NRMP Applicant Survey by Preferred Specialty and Applicant Type. Available at:https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2019/06/Applicant-Survey-Report-2019.pdf. Accessed August 2, 2021.

        View in Article

      5. Accreditation Council for Graduate Medical Education (ACGME. Common Program Requirements (Residency). Available at:https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResidency2020.pdf. Accessed August 2, 2021.

        View in Article
        • Brummond A
        • Sefcik S
        • Halvorsen AJ
        • et al.
        Resident recruitment costs: a national survey of internal medicine program directors.
        Am J Med. 2013; 126: 646-653https://doi.org/10.1016/j.amjmed.2013.03.018
        View in Article
        • Antono B
        • Willis J
        • Phillips Jr, RL
        • Bazemore A
        • Westfall JM
        The price of fear: an ethical dilemma underscored in a virtual residency interview season.
        J Grad Med Educ. 2021; 13: 316-320https://doi.org/10.4300/jgme-d-20-01411.1
        View in Article

      8. Association of American Medical Colleges (AAMC). ERAS Statistics 2016-2020- Internal Medicine (Categorical). Available at: https://www.aamc.org/data-reports/interactive-data/eras-statistics-data. Accessed August 16, 2021.

        View in Article

      9. Association of American Medical Colleges (AAMC). Fees for ERAS Residency Applications. Available at:https://students-residents.aamc.org/applying-residencies-eras/fees-eras-residency-applications. Accessed September 13, 2021.

        View in Article
        • Angus SV
        • Williams CM
        • Stewart EA
        • Sweet M
        • Kisielewski M
        • Willett LL
        Internal medicine residency program directors' screening practices and perceptions about recruitment challenges.
        Acad Med. 2020; 95: 582-589https://doi.org/10.1097/ACM.0000000000003086
        View in Article
        • Sweet ML
        • Williams CM
        • Stewart E
        • et al.
        Internal medicine residency program responses to the increase of residency applications: differences by program type and characteristics.
        J Grad Med Educ. 2019; 11: 698-703https://doi.org/10.4300/JGME-D-19-00194.1
        View in Article
        • Marshall AL
        • Halvorsen AJ
        • Kearns L
        • Reich LM
        • West CP
        • Oxentenko AS
        Disparities in salary and work-life integration in internal medicine program directors are associated with gender and partner employment status.
        Am J Med. 2021; 134: 285-291.e1https://doi.org/10.1016/j.amjmed.2020.10.004
        View in Article

      13. Qualtrics. Qualtrics Software. Version XM. Provo, UT: Qualtrics; 2020.

        View in Article
        • Stata Corp
        Stata Statistical Software. Version Release 16.
        StataCorp LLC, College Station, TX2019
        View in Article

      15. US Census Bureau. Census Regions and Divisions of the United States. Available at: https://www2.census.gov/geo/pdfs/maps-data/maps/reference/us_regdiv.pdf. Accessed December 1, 2020.

        View in Article

      16. Accreditation Council for Graduate Medical Education (ACGME). Accreditation Database System Online (Public). Available at: https://apps.acgme.org/ads/Public/Programs/Search. Accessed June 1, 2020.

        View in Article

      17. American Medical Association (AMA). Fellowship and Residency Electronic Interactive Database Access System (FREIDA) Online. Available at: https://freida.ama-assn.org/search/list?spec=42771. Accessed September 1, 2020.

        View in Article

      18. American Board of Internal Medicine (ABIM).Residency Program Pass Rates [2008-2019]. Available at:https://www.abim.org/Media/lhgmdidp/residency-program-pass-rates.pdf. Accessed June 2020.

        View in Article

      19. National Resident Matching Program. 2021 Applicant and Program Director Survey Findings: Impact of the Virtual Experience on the Transition to Residency. Available at:https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2021/05/Research-Brief-Virtual-Experience-2021-FINAL.pdf. Accessed August 17, 2021.

        View in Article

      20. Association of American Medical Colleges (AAMC). Supplemental ERAS application (for the ERAS 2022 cycle). Available at: https://students-residents.aamc.org/applying-residencies-eras/supplementalerasapplication. Accessed September 7, 2021.

        View in Article
        • Huppert LA
        • Santhosh L
        • Babik JM
        Trends in US internal medicine residency and fellowship applications during the COVID-19 pandemic vs previous years.
        JAMA Netw Open. 2021; 4e218199https://doi.org/10.1001/jamanetworkopen.2021.8199
        View in Article

      22. D'Angelo JD, D'Angelo AD, Mathis KL, Dozois EJ, Kelley SR. Program director opinions of virtual interviews: whatever makes my partners happy [e-pub ahead of print]. J Surg Educ. doi:10.1016/j.jsurg.2021.04.008. Accessed August 17, 2021.

        View in Article
        • Donahue LM
        • Morgan HK
        • Peterson WJ
        • Williams JA
        The carbon footprint of residency interview travel.
        J Grad Med Educ. 2021; 13: 89-94https://doi.org/10.4300/jgme-d-20-00418.1
        View in Article

      24. Society of University Otolaryngologists. Otolaryngology Preference Signaling. Available at: https://opdo-hns.org/mpage/signaling. Accessed August 18, 2021.

        View in Article
        • Hammoud MM
        • Standiford TC
        • Carmody JB
        The 2020-2021 residency application cycle: Lessons learned and lingering problems.
        JAMA. 2021; 325: 2249-2250https://doi.org/10.1001/jama.2021.5708
        View in Article
        • Nwora C
        • Allred DB
        • Verduzco-Gutierrez M
        Mitigating bias in virtual interviews for applicants who are underrepresented in medicine.
        J Natl Med Assoc. 2021; 113: 74-76https://doi.org/10.1016/j.jnma.2020.07.011
        View in Article

      27. Heitkamp NM, Snyder AN, Ramu A, et al. Lessons learned: applicant equity and the 2020-2021 virtual interview season. Acad Radiol. doi:10.1016/j.acra.2021.08.005. Accessed September 29, 2021.

        View in Article

      28. Hamade N, Bhavsar-Burke I, Jansson-Knodell C, et al. Virtual gastroenterology fellowship recruitment during COVID-19 and its implications for the future [e-pub ahead of print]. Dig Dis Sci. doi:10.1007/s10620-021-07014-1. Accessed August 17, 2021.

        View in Article
        • Hill MV
        • Ross EA
        • Crawford D
        • et al.
        Program and candidate experience with virtual interviews for the 2020 Complex General Surgical Oncology interview season during the COVID pandemic.
        Am J Surg. 2021; 222: 99-103https://doi.org/10.1016/j.amjsurg.2020.11.007
        View in Article
        • Jimenez AE
        • Khalafallah AM
        • Romano RM
        • et al.
        Perceptions of the virtual neurosurgery application cycle during the coronavirus disease 2019 (COVID-19) pandemic: a program director survey.
        World Neurosurg. 2021; 154: e590-e604https://doi.org/10.1016/j.wneu.2021.07.078
        View in Article

      31. Rajesh A, Asaad M, Elmorsi R, Ferry AM, Maricevich RS. The virtual interview experience for MATCH 2021: a pilot survey of general surgery residency program directors [e-pub ahead of print]. Am Surg. doi:10.1177/00031348211038555. Accessed September 30, 2021.

        View in Article
        • Rhoades JS
        • Ramsey PS
        • Metz TD
        • Lewkowitz AK
        Maternal-fetal medicine program director experience of exclusive virtual interviewing during the coronavirus disease 2019 pandemic.
        Am J Obstet Gynecol MFM. 2021; 3100344https://doi.org/10.1016/j.ajogmf.2021.100344
        View in Article

      33. Geary AD, Wang TS, Lindeman B, et al. Perspectives on virtual interviews-a follow-up study of the Comprehensive Endocrine Surgery Fellowship interview process [e-pub ahead of print]. Surgery. doi:10.1016/j.surg.2021.03.069. Accessed September 30, 2021.

        View in Article

      Article Info

      Publication History

      Published online: November 06, 2021

      Footnotes

      Funding: None.

      Conflicts of Interest: None. A summary of the results was presented during a freely available webinar for members of the Alliance for Academic Internal Medicine on August 5, 2021.

      Authorship: All authors had access to the data and a role in writing this manuscript.

      Identification

      DOI: https://doi.org/10.1016/j.amjmed.2021.10.017

      Copyright

      © 2021 Elsevier Inc. All rights reserved.

      ScienceDirect

      Access this article on ScienceDirect
      We use cookies to help provide and enhance our service and tailor content. To update your cookie settings, please visit the for this site.
      Copyright © 2022 Elsevier Inc. except certain content provided by third parties. The content on this site is intended for healthcare professionals.


      RELX