Note: The Q&A transcription has been paraphrased to be more succinct.
The recorded presentation may be viewed here.
Presentation: Variability in Data Presentation in the Medical Student Performance Evaluation (MSPE)
Speakers:
- Judith Brenner, MD, Associate Dean of Curricular Integration and Assessment, Donald and Barbara Zucker SOM at Hofstra/Northwell
- Karen Friedman, MD, MS-HPPL, FACP, Vice Chair for Education, Professor of Medicine, Donald and Barbara Zucker SOM at Hofstra/Northwell
- Jason Brenner, Research Associate and College Student, University of Michigan
Q: (Johmarx Patton)
In terms of building the extraction tool, what language did you use and how long did it take to build it?
A: (Jason Brenner)
The tool was not built in a specific computer programming language. It was mostly manual and recorded in an excel file. It took a couple weeks to set all the standards and build it, and then a couple more weeks to execute.
Q: (Johmarx Patton)
From the data that you pulled, at a quantitative level, are you able to compare between schools?
A: (Karen Friedman)
There’s really no way to compare amongst schools. There are very specific components of the MSPE that all program directors look at, but many of the schools come to the grades in a different fashion. For example, somebody receives honors on a medicine clerkship in one school and somebody receives honors on a medicine clerkship in another school. We know that both of those students received the best grades for those schools, but there are different components making up the grades at each school. It is the same for the percentiles and the overall comparatives. A medical school compares students within that school, but cannot compare students amongst different schools.
Q: (Amy Bingenheimer)
While the MSPE guidelines were helpful as a start, why can't there be some additional standards established for healthcare professions? Or does that need to wait for a competency-based curriculum?
A: (Judy Brenner)
That is the question of the day, and I think it’s a really important one. It will really be up to UME and GME communicating with one another about what is most useful. I think Karen’s last slide really describes the continuum of medical education. Every school collects a ton of data. Residency programs collect a ton of data. It will involve coming to a consensus of what is most important, and understanding that to a different program a different set of data may be most relevant. I think that’s where we are headed. But, more conversation between the different stakeholders is where I personally think that we should be headed.
Q: (Tori Flores)
As a new medical school working on building our MSPE, what tips could you give us?
A: (Judy Brenner)
As a relatively new medical school, we matriculated our first students in 2011; we had our first graduates in 2015. From the beginning, we used the MSPE guidelines pretty thoroughly. We paid a lot of attention to the recommendations when they came out, which was early on in our time of building MSPE. But we also really tried to be true to our assessment system. When building our MSPE we always used the principle of trying to be true to the system and as transparent as we could possibly be to our audience, which were our program directors. That would be the advice I would give to you: stay true to the systems that you’ve built, but I would hope people would feel comfortable and confident about displaying that to the world. We thought as a school we needed to be as clear as possible, so we keep iterating ours a bit to really try to enhance clarity.
Q: (Terence Ma)
Sounds like grades are arbitrary and thus not useful. How would you want the data to show? What data do you want?
A: (Karen Friedman)
I don’t think any us believe that grades are arbitrary or useless. I think schools spend a lot of time setting up their assessment standards and the grades they’re putting forward. The issue we have is that the way schools come about giving grades is different, and that is really where the problem lies. It is not that the grades are useless. In fact, some schools are pass/fail. It just makes it difficult for the end-user, the program director, to evaluate different students from different schools. What we are suggesting are more uniform and agreed-upon components that make up the specific grades by which we are evaluating students across schools.
Q: (Alison Holmes)
Like in most performance evaluation, only 5-10% of students are truly outstanding, and probably only 5% are potential "problems." If there are 80% of applicants in the middle that are not really all that distinguishable but will make competent physicians, why all the effort to make fine distinctions?
A: (Judy Brenner)
This is a very good question. I think it is done to make it simpler for the reader of the MSPE to decide on candidates for residency. As we said, the number of applicants and the amount of information to review in deciding on candidates to interview and eventually rank. The comparable performance indicator makes it a bit simpler for some. However, your question is on target because assigning an adjective or quintile often dilutes the complexity and elegance of assessment data that schools might have. Personally, I would love to see a system where medical schools could share competency-level data that showcases areas of strength as well as areas to improve on.
Q: (Rajesh Mangrulkar)
How much weight do program directors put on Step scores and other national exams?
A: (Karen Friedman)
All programs put a different amount of weight on the Step scores and other national exams based on the specific residents they are looking for.
Presentation: A Generative Case System That Reflects Our Patient Population
Speakers:
- Mike Paget, Manager, Academic Technologies, University of Calgary, Cumming School of Medicine
- Scott Steil, Developer, University of Calgary, Cumming School of Medicine
Q: (Johmarx Patton)
In terms of sharing the cases have you been utilizing the virtual patient standard or another standard for sharing the data?
A: (Mike Paget)
That is a great question. The ANSI virtual patient standard didn’t have a mechanism that allowed us to capture “fuzzy,” as it were. If someone wanted to transport their patient content, we would be happy to do so - and even consider writing an export that would export an instance to the ANSI virtual patient standard. But, having spent years of my life on virtual patient, no one ever showed up with a virtual space patient in that standard and wanted to share that content.
Working in OpenLabyrinth we found that the work I was doing five or six years ago was often trying to help get virtual patient servers set up on institutional platforms or systems. We’ve come a long way. The approach taken with national curriculum bodies in Canada – with both pediatricians and standard medical practitioners – has been to host that content and then give institutions the ability to pull student reports back out. The sharing of the content never seemed to happen, rather people really seemed to want to be able to send students through an already up-and-running system, do this formative content, and at times have a report back. That was a bit of a long-winded answer. If people came calling with the ANSI spec, I think we’d rise to the occasion, but it just doesn’t seem to be part of the ecosystem of medical education to share this material much.
Q: (Victoria Richards)
Is there any audio component to the cards to reflect diversity of language?
A: (Mike Paget)
That is a great question. We do have some ultrasim sound cards that have brief video clips. Scott, can we put an audio clip in there?
A: (Scott Steil)
Yes, we can support audio clips, though the question also mentions diversity of language. To some extent, we have the ability to randomize the audio clip and to randomize the video clip, but not such that it would support different languages.
A: (Mike Paget)
Canada has two official languages. A point of growth for us is to potentially have a French version of the website. It’s challenging, however, to write good, re-playable cases in multiple languages; I think it would always likely stand as separate cases.
That is also a great question in terms of physician-patient communication. At the University of Calgary Cumming School of Medicine, one of the strengths is a robust medical skills course with deep communication. We never saw ourselves trying to recreate patient-doctor communication by this system. We have other mechanisms for doing it. Previously in Unity and Second Life, we tried really hard to have natural language processing. It was really hard to do. Every time a new cohort of students attempted one of those cases, they would find more holes than gold. It’s a great question, and it’s a really challenging space. One I would say we can answer to some extent by this system, but answer better with standardized patients.