A competency-based clinical assessment tool available in real time on a phone application may be preferred to traditional time-in-training assessments by both learners and assessors.
ABSTRACT
Introduction: To align with competency-based medical education, Emory University’s Anesthesiology Residency program switched from evaluation using a traditional time-in-training assessment to the Entrustable Professional Activity (EPA) framework of myTIPreport.
Methods: The study compared the use of the competency-based myTIPreport tool from January to December 2023 to the time-in-training model utilized from January to December 2022. Effectiveness of each tool was evaluated by user surveys, evaluation of usage data, and faculty engagement.
Results: There were 51 residents per year (75 residents in total) and 185 participating faculty members. Compared to the previous tool, myTIPreport was preferred by residents (86%) and faculty (58%), and was reported as significantly more satisfying, easier to use, and useful. Assessments submitted increased 6.5 times from 249 to 1,625 assessments per year. Ninety-five percent of residents had more assessments completed during this time, with a median 14 assessments per resident per six months compared to 1.5 using the old tool. Faculty participation increased from 24% to 76%, and the median assessments submitted per faculty member per year doubled from three to six (P<0.001 for all comparisons except faculty preference [p=0.032]).
Conclusion: Transitioning to an EPA-based framework for a competency- and workplace-based assessment of resident performance utilizing myTIPreport improved engagement by Department of Anesthesiology faculty, increased the number of assessments per resident, and was preferred by both residents and faculty compared to the pre-existing, time-in-training model.
INTRODUCTION
Though the Accreditation Council for Graduate Medical Education (ACGME) has supported the concept of competency-based medical education (CBME), full operationalization and implementation of a CBME system within anesthesiology residency programs has yet to be achieved (Ebert and Fox, 2014). CBME operates on a core set of principles: that competencies are role-specific, that assessment is predominantly performance-based and directly tied to specific competencies, that individuals advance through training based on demonstrated mastery of competencies, and that instructors tailor feedback to the competency or competencies observed in their learners (Long, 2000). Within this paradigm, education programs focus on desired learner outcomes rather than program structure (Carraccio et al., 2016).
Through utilization of a core components framework (Van Melle, et al., 2019), the anesthesiology residency program at the Emory School of Medicine identified trainee assessment, specifically the need for a programmatic approach to competency-based trainee assessment, as an area to target for improvement.
Our prior assessment tool, New Innovations (New Innovations, Uniontown, Ohio), is a web-based multiple-choice questionnaire with 30 questions regarding a resident’s performance, with room for narrative comments. The system uses a “time in training model” in which evaluations are completed at fixed times and has two flaws that make it unsuitable for competency-based assessment. First, the platform was not designed for frequent low-stakes feedback, and second, feedback is not easily tailored to specific competencies. To address these issues, our team transitioned to myTIPreport (Training for Independent Practice) (myTIPreport, 2026), a mobile and desktop assessment tool designed specifically for anesthesiology training that facilitates immediate, targeted feedback (Woodworth et al., 2024). Assessments can be triggered by residents who can request them of their attending physicians at any time, or independently provided by faculty. Assessments are focused on a particular skill or procedure and are efficient, taking less than 1-2 minutes to complete. (Details of both tools provided in Appendix 1A & 1B).
The myTIPreport platform offers multiple channels for assessment; for daily workplace-based feedback, faculty can use an Entrustable Professional Activities (EPA) framework which allows objective assessment of trainees’ achievement of independence performing specific clinical activities and procedures such as such as “airway assessment and management” or “regional pain management”. Assessors are asked to provide formative feedback and to rate residents’ performance within an EPA on a 5-point scale with 1 indicating that the resident requires oversight and 5 representing the ability to perform that activity or procedure independently. The system tracks resident performance longitudinally and is readily available for resident and faculty review, offering valuable insights into growth as well as opportunities for reflection and improvement. Program directors can further utilize myTIPreport data to implement benchmarks for advancement in resident training. The system has the added advantage of being embedded in a smart phone application, reducing the barrier to accessing the system and making the process much more efficient.
One concern about our previous evaluation system was the small number of assessments that each resident received from individual faculty members, and we chose this indicator to focus on as we implemented myTIPreport. We hypothesized that given the ease of use, targeted feedback options, and seamless integration with a mobile application, residents and faculty would find the myTIPreport assessment tool more useful and would prefer it over the old system, thus increasing the proportion of faculty engaged in writing assessments, the number of assessments written per faculty member, and the number of assessments completed per resident.
METHODS
This study was conducted within the Emory School of Medicine’s Department of Anesthesiology residency, a four-year training program with twelve residents per year. The residents rotate through multiple healthcare locations, including Emory University Hospital, Emory University Hospital Midtown, Grady Memorial Hospital, Children’s Hospital of Atlanta, and others. The study was approved by Emory’s Institutional Review Board, Study 000052379.
To introduce the tool to faculty, we initiated a “soft role out” in late 2022, in which the tool was not required to be used for assessment, but instead made accessible for faculty to familiarize themselves with it. In 2023, we held two department-wide grand rounds sessions on CBME and EPAs to educate our faculty and increase engagement. In January 2023, a department-wide email was sent out with training videos on assessment and a “Frequently Asked Questions” dialogue to all faculty detailing our new tool and the rationale behind the transition. At monthly didactics and through email reminders, residents were encouraged to use the new system to ask their attendings for feedback.
The myTIPreport tool was evaluated from January to December 2023 and compared to New Innovations time-in-training assessment data from January to December 2022. We surveyed faculty and residents on their perceptions of the old tool in November 2022 and myTIPreport in May 2025. Quantitative comparison of the old and new tools was based on the relative number of total assessments submitted, the median number of assessments per resident, the number of faculty submitting assessments, and the median number of assessments submitted per faculty member.
In addition, we developed resident and faculty satisfaction surveys using Likert scales to evaluate the perceived usefulness and barriers to use of each assessment tool. The short survey was administered using Research Electronic Data Capture (REDCap), hosted at Emory University. REDCap is a secure, web-based software platform designed to support data capture for research studies (Harris et al., 2009; Harris et al., 2019). The survey for the old system asked three questions of attendings and residents; the first asked residents if they were satisfied with feedback that it provided and asked faculty if the tool was easy to incorporate into their workflow. The second question asked if the tool had been useful for receiving or providing feedback, respectively. The last question for both groups asked what barriers they saw to the use of the tool, with 6-7 options, including time to complete, difficulty of use, lack of value to learner or faculty, improper framework, or desire to avoid giving negative feedback (faculty only) and other barrier, with an open-text response.
The myTIPreport survey had the same three questions, plus two more. The first asked which tool they preferred, if they had also used the old tool. The second asked an open-ended question for their comments and suggestions about myTIPreport: its use, limitations, comparison to the old tool, etc. Exact wording of the survey questions is in Appendix 2.
The study sample size was fixed based on the number of anesthesiology residents accepted at Emory. Therefore, a sensitivity power analysis was conducted using the G*Power calculator version 3.1.9.4 (Faul et al., 2007). With two tails, normal parent distribution, alpha threshold to reject the null hypothesis at 0.05, a power of 80% and 51 residents per group, a 2-group Wilcoxon-Mann-Whitney U Test would be able to detect a “medium-large” effect size of d=0.57. We felt confident that the effect size we would see would be larger than this, so proceeded forward with the study.
Data were analyzed using SAS 9.4 (Cary, NC). Statistical analysis used non-parametric tests for continuous outcomes, so outlier values did not have undue influence on results. The Mantel-Haenszel Chi-Square Test was used on the Likert scale survey questions, as ordinal data.
We analyzed the performance of each tool by dividing the data into three broad categories. First, we measured user engagement by comparing the total number of assessments completed per year for each tool, the number of assessments per resident per six months, the proportion of assessors using each tool per year, and the number of assessments completed per assessor per year among those who completed any assessment. Second, we examined the perceived utility of each tool by comparing responses from the survey questions, including satisfaction, usefulness, barriers to use and direct preference from both residents and faculty. Lastly, we analyzed qualitative, open-text responses in the survey by grouping responses into themes, to identify user issues that were not included in the preset survey responses. Table 1 shows the specific statistical test used for each outcome.
RESULTS
There were 12 residents per year for the 4-year program, with 15 for the Class of 2023. This is 48-51 participating residents per year, or 75 unique residents. Residents were not assessed during their first semester. There were 185 participating faculty members for both years. Three assessments were excluded using the new tool from one assessor who left in the middle of 2023. One assessment was excluded from another assessor who was a surgeon, not an anesthesiologist. Finally, five assessments were excluded, all using the new tool, completed for two oral and maxillofacial surgery residents.
The implementation of myTIPreport resulted in a dramatic growth in the total number of assessments submitted over the course of the year. From January to December 2022, 249 assessments were submitted using our previous tool. From January to December 2023, 1,625 assessments were submitted using myTIPreport representing a 6.5-fold increase in the number of submitted assessments (p<0.001) (Table 2).
This was paired with a dramatic increase in the median number of assessments per resident. Within a six-month window, on the previous assessment platform, the median number of assessments per resident was 1.5, compared to a median of 14 using myTIPreport. Furthermore, residents who were present both for when the old tool was used and when myTIPreport was used experienced a median growth of ten assessments per six-month window (Table 2, Figure 1, p<0.001 for both comparisons). Additionally, among residents, 8% had at least one assessment with myTIPreport completed but none with the old tool, while an additional 87% had more assessments completed, or 95% of residents with an increased number.
Figure 1: Number of assessments per resident per 6 months.
A. Residents had significantly more assessments completed per 6 months using the new tool (median 14, IQR 8.5-22) than the old tool (median 1.5, IQR 1-3.5) (p<0.001, Mann-Whitney U Test). Data are presented per 6 months to account for residents who were not present for the entire calendar years of 2022-2023. First semester residents are excluded; new residents are not routinely assessed during this period. The rectangles of the boxplots reflect the Q1, median and Q3 of the data, the diamond the mean, and circles beyond the whiskers, outliers of the data.
B. Difference in number of assessments completed per resident using myTIPreport versus the old tool (median 10 more assessments, IQR 4.5-18.5) (p<0.001, Signed Rank Test). The 39 residents exclude n=24 residents not present for 2022 or 2023, as well as n=12 residents who joined in 7/2022. New residents are not routinely assessed during their first semester.
Faculty submission of resident assessments was another area where we saw a large change in behavior after the implementation of myTIPreport. Forty-five out of 185 faculty members submitted assessments using the previous tool compared to 140 faculty members who submitted assessments using myTIPreport, representing a three-fold increase in participation from 24% to 76% (p<0.001). Along with the greater percentage of faculty members completing assessments using myTIPreport, assessors using the new tool completed significantly more assessments. The median assessments per faculty member grew from three to six, with the percentage of assessors completing five or more assessments growing from 35% to 55%. Faculty members who used both tools submitted a median of four more assessments using myTIPreport. Among assessors using any tool, 67% had not used the old tool but started using myTIPreport, while an additional 26% completed more assessments than they had with the old tool; together, 93% showed an improvement in use (Table 1, Figure 2, p<0.001).
Figure 2: Number of assessments per assessor by tool
A. Assessors who used the tools completed significantly more assessments using myTIPreport (median 6, IQR 2-15) than the old tool (median 3, IQR 1-10) (p<0.001, Mann-Whitney U Test). Rectangles of the boxplot reflect the Q1, median and Q3 of the data, the diamond the mean, and circles beyond the whiskers reflect outliers.
B. Among assessors using myTIPreport (n=140), 55% completed five or more assessments for the year, compared to 35% of assessments using the old tool (n=45).
C. Difference in number of assessments completed per assessor.
For the surveys, resident response rates were 57% and 41% for the old tool and myTIPreport, respectively. Faculty response rates were 19% and 23%, respectively. The myTIPreport survey was completed in a median 1.2 minutes.
Survey data supports the success of myTIPreport over the old tool. Among residents, almost three-quarters (72%) either strongly agreed or agreed that they were satisfied with myTIPreport, while only a quarter (24%) felt the same about the old tool (p<0.001). Additionally, over three-quarters (81%) strongly agreed or agreed that myTIPreport was useful compared to 31% for the old tool (p<0.001). A similar trend was observed in faculty perceptions of the transition. Only 8% of faculty strongly agreed or agreed that the old tool was easy to use and 19% that was useful compared to 43% and 62% respectively with myTIPreport (both p<0.001). More than three-quarters (78%) of faculty disagreed or strongly disagreed that the old tool was easy to use. This dropped to 21% with myTIPreport (p<0.001). Overall, the vast majority of residents (86%) and a majority of faculty (58%) who had used both tools expressed a preference for myTIPreport over the old tool with p<0.001 and p=0.032 respectively (Figure 3).
Figure 3 Survey Evaluation of Assessment Tools
A. Survey responses comparing resident and faculty perceptions of the old tool (New Innovations) versus myTIPreport. Responses are shown for satisfaction (residents), usefulness (residents and faculty), and ease of use (faculty). Statistically significant improvements were observed in each category for myTIPreport across all groups. Exact wording of the questionnaire is available in Appendix Table A2.
B. A majority of residents (86%) and faculty (58%) preferred myTIPreport over the old tool (p < 0.001 and p = 0.032, respectively). “Did not use the old tool” responses (n=7 and n=4) were omitted from the analysis.
More than half of both residents and faculty felt that difficulty of access and time to complete were barriers to utilizing the old tool. Half of faculty felt the old tool lacked value to faculty. A third or more of both residents and faculty felt the old tool lacked value to the resident and was an improper framework for assessment. A third of faculty (36%) also felt that the desire to avoid giving negative feedback was a barrier. For myTIPreport, about the same percentage of faculty felt that the tool was an improper framework for assessment (42% vs. 39% for the old tool), while a fifth of residents (19%) felt the same, compared to 31% for the old tool. A quarter of residents and faculty (29% and 24%) listed time to complete as a barrier, down from more than half (52% and 61%) for the old tool. A fifth of residents and faculty felt myTIPreport lacked value to faculty (Figure 4, Appendix Table A4-A5).
Figure 4 Barriers to participation
Barriers to using the old tool (blue; New Innovations) and myTIPreport (purple) as reported by residents (left) and faculty (right). The most frequently cited barriers for the old tool included difficulty accessing the system and time required to complete evaluations.
For the myTIPreport survey only, respondents were given the opportunity to give additional comments (Appendix Table A6-A7). Many of these comments were echoed in the Other Barriers listed (Appendix Table A4-A5). Some residents wanted the ability to automatically submit feedback or wanted quicker feedback. Some had trouble navigating to specific skills or otherwise found it cumbersome with too many options. Some thought the quality of feedback was often “too positive” (i.e., not constructive) or otherwise not useful. One reported that the app crashed frequently for them.
Among faculty, some commented that residents did not request assessments frequently or only requested them when they felt they performed well. Some found myTIPreport difficult to navigate or to find a specific skill. Some questioned the utility of the feedback given, with one stating that while it assesses technical competence, it does not assess “big picture” issues or nuance well, such as communication with other teams or triaging multiple emergencies. Some commented on the difficulty of providing negative feedback in a non-anonymous forum. Several commented how much they prefer myTIPreport over the old tool.
DISCUSSION
Implementing myTIPreport resulted in significant increases in user satisfaction, ease of feedback, and perceived utility of feedback. This facilitated increases in total assessments submitted, assessments received per resident, faculty participation in resident assessment, and assessments completed per faculty member, furthering the department’s goal of a clear system of resident assessment.
Survey data indicated that faculty and residents found engaging with myTIPreport to be easier, faster, more useful, and more valuable than the old tool. Residents commented that specific highlights of the program were their ability to track growth over time and to implement changes based on their targeted feedback. Faculty, however, expressed a less dramatic difference in preference compared to residents. The most commonly identified weaknesses that help to explain this are in workflow integration, feedback quality, and feedback reception. Faculty do not work with residents every day, thus integrating this new system into their workflow was difficult.
Challenges with delivering negative feedback was another commonly identified theme. As requesting an assessment is largely resident-driven, faculty felt that residents were hesitant to request evaluations from faculty with whom they had “bad days.” Furthermore, faculty noted in the qualitative comments that since feedback was immediately delivered to residents, there was a hesitancy to deliver negative feedback and that this tool serves as a complement to, rather than a replacement for, verbal feedback. Lastly, when soliciting feedback from faculty using MyTIPreport, residents are prompted to provide feedback back to the evaluator. However, faculty identified a decrease in the verbal feedback that they receive from residents and expressed concerns that this inhibited both improving their own practice and facilitation of academic advancement.
Despite these weaknesses, our findings demonstrate a dramatic positive change in measurable behavior of faculty regarding resident assessment. The implementation of this tool accomplished the goal of improving the number of low-stakes assessments that residents achieve, inching our program closer to a more ideal system of programmatic assessment. This steadier stream of feedback will allow us to advance our efforts further to tailor the learning experience of any given resident to the uniqueness of their educational needs and ensure equity of experience for our learners.
The EPA framework has also helped our residents and attendings to be more aligned with expectations and requirements; having a shared mental model empowers residents to seek more meaningful feedback from attendings and facilitates attendings efforts to provide targeted and actionable feedback to residents. This system seems to provide our residents with incentive to solicit feedback both to improve their practice and demonstrate their mastery. Since feedback is not required at any specific interval, we have found the flexibility of this system to be very accommodating to our current wellness efforts; residents can accrue the assessments to demonstrate their progress towards competency at their own pace.
We recognize that providing our faculty with a tool that improved the speed and ease of submitting an assessment likely had an impact on their behavior, possibly as much as any other. Additionally, the resident education leadership took several steps to prepare faculty for this transition, and it is undeniable that these efforts contributed to the results we measured.
Study limitations include that this is the experience of a single institution. Results may not be universally generalizable, particularly if existing assessment models for residents elsewhere differ significantly from what we previously used. Additionally, while increased engagement by assessors is valid, a direct comparison of assessments completed per resident per semester may not be completely fair, as there is an expectation to complete more of them under myTIPreport than under the old tool. Further, the study did not directly evaluate the quality and effectiveness of assessments generated by the new tool.
In addition, this study only evaluated myTIPreport, and no other CBME model tools which may be equally effective or have other advantages. Lastly, the post-implementation survey was performed in May 2025, which is further from the implementation than ideal and potentially resulted in missed resident data. Further evaluation of myTIPreport would help to gauge the accuracy and applicability of the data that it provides in comparison to that of the previous platform.
While the total number of assessments may be an imperfect outcome, the dramatic increase in faculty utilization and engagement and the comparison of survey data suggest the success of myTIPreport in addressing these issues and providing valuable data for all users.
Our program still has progress to make in terms of compliance with daily assessments; to this end, we aim to provide continued education and emphasis to both residents and faculty on the benefits of feedback. Further studies could evaluate the effect of feedback-focused professional development on faculty perceptions of this tool. We also recognize that although we have improved one aspect of our approach to trainee assessment, we have much more work to do in this arena to develop a more comprehensive approach to assessing our trainees’ progress to independent practice.
CONCLUSION
In conclusion, transitioning from a time-in-training model to myTIPreport to assess anesthesiology residents not only resulted in a dramatic increase in total evaluations submitted and received per resident, it also resulted in dramatically wider and deeper engagement amongst faculty. This profound increase in evaluation was driven by increased satisfaction, ease of use, and utility of myTIPreport as well as addressing previously perceived barriers to using the prior tool.
ACKNOWLEDGMENTS
The authors received no grant support for this work and declare they have no conflicts of interest in regard to this work.
Carraccio, C., Englander, R., Van Melle, E., ten Cate, O., Lockyer, J., Chan, M. K., Frank, J. R., Snell, L. S., & Collaborators International Competency-Based Medical Education. (2016). Advancing competency-based medical education: A charter for clinician-educators. Academic Medicine, 91(5), 645–649. https://doi.org/10.1097/ACM.0000000000001048
Ebert, T. J., & Fox, C. A. (2014). Competency-based education in anesthesiology: History and challenges. Anesthesiology, 120(1), 24–31. https://doi.org/10.1097/ALN.0000000000000039
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146
Harris, P. A., Taylor, R., Minor, B. L., Elliott, V., Fernandez, M., O’Neal, L., McLeod, L., Delacqua, G., Delacqua, F., Kirby, J., Duda, S. N., & REDCap Consortium. (2019). The REDCap consortium: Building an international community of software platform partners. Journal of Biomedical Informatics, 95, 103208. https://doig.org/10.1016/j.jbi.2019.103208
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009). Research electronic data capture (REDCap): A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2), 377–381. https://doi.org/10.1016/j.jbi.2008.08.010
Long, D. M. (2000). Competency-based residency training: The next advance in graduate medical education. Academic Medicine, 75(12), 1178–1183. https://doi.org/10.1097/00001888-200012000-00009
myTIPreport, LLC. (n.d.). myTIPreport: Simple, real-time feedback platform for medical schools, residencies, and fellowships. Retrieved September 5, 2026, from https://mytipreport.org/
Van Melle, E., Frank, J. R., Holmboe, E. S., Dagnone, D., Stockley, D., Sherbino, J., & Collaborators International Competency-Based Medical Education. (2019). A core components framework for evaluating implementation of competency-based medical education programs. Academic Medicine, 94(7), 1002–1009. https://doi.org/10.1097/ACM.0000000000002743
Woodworth, G. E., Goldstein, Z. T., Ambardekar, A. P., Arthur, M. E., Bailey, C. F., Booth, G. J., Carney, P. A., Chen, F., Duncan, M. J., Fromer, I. R., Hallman, M. R., Hoang, T., Isaak, R., Klesius, L. L., Ladlie, B. L., Mitchell, S. A., Miller Juve, A. K., Mitchell, J. D., McGrath, B. J., . . . Maniker, R. B. (2024). Development and pilot testing of a programmatic system for competency assessment in U.S. anesthesiology residency training. Anesthesia & Analgesia, 138(5), 1081–1093. https://doi.org/10.1213/ANE.0000000000006667
Woodworth, G. E., Marty, A. P., Tanaka, P. P., Ambardekar, A. P., Chen, F., Duncan, M. J., Fromer, I. R., Hallman, M. R., Klesius, L. L., Ladlie, B. L., Mitchell, S. A., Miller Juve, A. K., McGrath, B. J., Shepler, J. A., Sims, C., III, Spofford, C. M., Van Cleve, W., & Maniker, R. B. (2021). Development and pilot testing of entrustable professional activities for U.S. anesthesiology residency training. Anesthesia & Analgesia, 132(6), 1579–1591. https://doi.org/10.1213/ANE.0000000000005434
Ari Levine, MA
Medical Student, Emory University School of Medicine, ORCID: 0009-0004-4919-7390
David W. Boorman, MS
Senior Biostatistician, Department of Anesthesiology, Emory University School of Medicine, ORCID: 0000-0002-0889-6150
Michelle Romej, MD
Assistant Professor, Department of Anesthesiology, Emory University School of Medicine
Susan Smith, MD
Assistant Professor, Department of Anesthesiology, Emory University School of Medicine
Sona Arora, MD
Associate Professor, Department of Anesthesiology, Emory University School of Medicine
McKenzie Hollon, MD, FASE
Associate Professor, Department of Anesthesiology, Emory University School of Medicine
Published 11/2/26