The Subcommittees then designed the screening rubric and interview structure using the key qualities as the framework. Two orientation sessions were conducted to provide thorough walkthroughs of the updated screening and interview processes.

Screening – Development of the screening tool was centered around the qualities identified as predictive of success in the residency program. To reduce bias, the evaluations within the screening tool were intentionally created to be as objective as possible. The screening tool was divided into three sections. The first section included three qualification questions pertaining to grade point average (GPA) and submission of the supplemental application and transcripts. Applications that did not meet the pre-determined GPA cutoff or did not include the supplemental application or transcript were ineligible for review.

The second section of the screening tool focused on the character evaluations and narrative comments of the letters of recommendation.  The character evaluations scores were worded as “fails to meet,” “appropriate,” or “exceeds” for each characteristic. Reviewers were instructed to award or deduct points based on their assessments of the characteristics or narrative comments if explicit keywords related to the characteristic were mentioned.

The third section of the tool searched for our desired qualities through all other materials within the application including Curriculum Vitae , letter of intent, and supplemental materials. The supplemental materials consisted of two open-ended questions for candidates to answer. Guidance was provided to assist with scoring in this section.  A scoring tool was created using Microsoft Excel to tally up points from each of the three sections using a weighted model. This tool was internally validated based upon archived applications. The standard deviation between the validated scores were obtained to establish the threshold for bringing in a third reviewer, if necessary. More weight was given to qualities determined to be identifiable only from the screening process. Each candidate was randomly assigned to a pair of evaluators for scoring. Any score above the standard deviation required additional review and the average of the three scores were used. The final scores obtained from the screening tool were entered into the Pharmacy Online Residency Centralized Application Service website. 

Interview Selection Process and Meeting Structure – The papercut meeting included a broad group of preceptors that work with residents, and its primary objective was to select candidates who would be extended invitations for interviews and identify alternate candidates. To ensure a productive discussion, rankings were distributed to preceptors prior to the meeting. Candidates were ranked from highest to lowest based on screening scores. Particular attention was given to those ranked in the middle due to the very small differences in scores. Preceptors were expected to review their assigned applicants and be prepared to present them to the group. Following the presentation, other preceptors provided input and asked questions.

Prior to the deliberations, several ground rules were established. First, all reviewers agreed to trust the efficacy of the screening tool. Second, any discussion points were expected to be presented in a professional and respectful manner, promoting inclusivity of thought. Last, any perceived bias was promptly identified and addressed by the meeting facilitator or other attendees. During the meeting, each reviewer pair was allotted uninterrupted time to provide their insights on each of their candidates. Additional input could be provided by others with firsthand experience with a candidate.

A majority vote was required to retain or modify a candidate’s ranking. Elevating a candidate’s position could be justified by exceptional qualities, attributes, or experiences that might not have been fully captured by the screening tool. Conversely, lowering a candidate’s rank could be a result of significant red flags, untrainable qualities, misalignment with the values and strengths of the residency program, or deficiencies in desired attributes.

Interview Process – Interviews during the 2022 recruitment cycle were conducted virtually but were designed to be easily replicated in person. The process included an interview portion alongside two unique simulation activities.

The interview component consisted of questions targeting one or two of the qualities or skills determined by the recruitment committee. Interviewers were provided with an interview scoring rubric. Candidate responses were scored on a numeric scale with the option to select N/A if unable to assess an answer. Explicit criteria, keywords, and descriptive examples of answers that would meet the desired scoring in the scoring rubric were included.

To develop the new interview questions and scoring tool, the interview subcommittee reviewed and revised questions used in previous interviews to ensure their relevance, while also creating new questions to best assess the predetermined qualities. Additionally, any questions that demonstrated bias towards candidates with specific types of experiences were removed. Each candidate spent time on two virtual interview rooms, each focusing on five questions. This approach allowed candidates opportunities to ask questions and take breaks in between sessions.

The interview process also included two simulations, providing evaluators the opportunity to assess candidates’ interpersonal communication, critical thinking, and adaptability – qualities that may be challenging to evaluate in the traditional interview setting. In one simulation, focused on teamwork, candidates were randomly paired and Candidate One was provided a drawing and instructed to direct Candidate Two to draw the same item. The second simulation focused on assessing candidates’ ability to express empathy and passion for patient care through a mock patient counseling session. To separate the assessment from candidates’ clinical knowledge, fake drug names were used. 

DISCUSSION

Our previous approach to the pharmacy resident selection process was notably subjective and introduced bias due to a reliance on the titles of letter writers, research experience, school of pharmacy attended, and advanced pharmacy practice experiences. The former screening rubric lacked clarity and was based upon loose criteria with flexible point allocations during application review leaving applicants’ fates subject to individual reviewers’ interpretations.  We created a new process to mitigate these issues and to meet the new ASHP standards. While several resources provide guidance on following an unbiased recruitment approach, the committee did not find any published criteria or frameworks specific to residency recruitment and thus created its own.

The goal of this initiative was to optimize the residency recruitment process while reducing bias and potential discrimination. Knowing that candidates’ unique skills are better predictors of success in residency than their job-related experiences, the residency recruitment committee first defined the specific skills and qualities required to be successful within a pharmacy residency program. These skills and qualities were then distributed amongst the screening and interview processes based on where they could best be evaluated. The modified process provided preceptors with clear guidance and specific criteria for awarding points in both the screening and interview rubrics as well as in the rankings of candidates. The new interview process included two simulations and two interview panels, which allowed for the evaluation of candidates in real-life scenarios and in response to structured questions. The ranking meeting similarly benefited from structured discussions regarding candidate feedback and ranking criteria.

Although we were focused on minimizing bias in our processes, we did choose to continue to use a GPA cutoff in our initial review of candidates as it was part of the screening process in prior years, though notably, the cutoff was lowered. Because GPA has been correlated with higher match rate (Philips et al., 2016), the use of this criterion for candidate selection is controversial as it may limit recruitment of a diverse workforce (Maurer, 2020) and remains an area for further investigation.

The main limitation of our new framework is a lack of objective measures that highlight the transition of the residency selection process from a subjective, potentially biased process to a more objective process and identifies any aspects of the process that may still be more subjective in nature and prone to bias. One way to demonstrate this may be to evaluate the difference in the range of screening and interview scores between reviewers in the old and new processes to see if larger variance existed among reviewers prior to implementation of the new process. Next steps include an ongoing annual quality improvement process as well as applying this framework to the next residency recruitment cycle with a large selection committee and identifying variations in screening and interview scores per section to identify any aspects of the framework that may still be subject to more bias or interpretation. 

CONCLUSION

EUHM has undertaken a transformation in its pharmacy residency recruitment process to reduce biases and potential discrimination. Recognizing the imperative to reduce implicit bias and promote fairness, the residency recruitment committee made key improvements in the screening, interview, and selection processes. The development of a new screening tool, structured interviews, and simulations aimed to assess candidate qualities objectively. This updated process addressed previous subjectivity and elevated transparency, making the process more equitable. The impact of this work has influenced divisions within the pharmacy department to update their recruitment practices including other residency programs and clinical pharmacist interviews. Future steps involve measuring and evaluating the impact of these changes on bias and recruitment practices by the ability to reproduce the results with various selection committees on the same subset of candidates. The authors’ hope is this revised framework will serve as a tool for other institutions and healthcare disciplines to adopt to incorporate more objective criteria into their recruitment processes. 

ACKNOWLEDGMENTS

The authors received no grant support for this work and declare they have no conflicts of interest in regard to this work.