Let's start with a basic test and give it a patient vignette that any 3rd year med student should get right
Bravo! It correctly recognized Virchow's Triad for DVT: venous stasis, vascular injury, and hypercoagulability. Pretty cool for an open source AI. Let's try another soft ball
And again it nails the diagnosis. Pretty easy when the patient gets the highest score possible on the Centor Criteria. We should make it a little more difficult.
Oh boy.... I'd let a med student and maybe a 1st year resident get away with this answer but it's clearly missing the big picture.
Perhaps the temperature on the AI model is too low, let's increase it to 0.66 and see if that helps it understand
I'm still an unhappy attending. I wondered if it was my wording of the heart rhythm so I added "irregularly" irregular to really try and hammer the point home but it didn't help the poor trainee. So I gave it an EKG report:
Eh, it's closer but still not fully seeing the patient's issues. The young clinician has some work to do. COPD, age, and OSA are independent risk factors for atrial fibrillation due to stress on the right atrium. Rapid a-fib leads to CHF. It's not clear the trainee sees this
Let's test it's ability at differential diagnosis. We'll start with a classic
That was shockingly good for an AI. I'm truly impressed. Let's see if it can change the DDx with some tweaks in the vignette
Nice. We added musculoskeletal, GI, pulmonary, and psychiatric causes of chest pain! ChatGPT also lowered the odds of catastrophic disease with the improved vital signs. Maybe this M1 has brains... Let's throw it a curveball and see if it has a zebra rifle
Swing and a miss. You need to review your endocrine textbook. But as the attending of an overworked trainee I take pity on you. I'll order some labs and see if you can figure it out. I know you'll kick yourself the rest of your career if you miss this one:
I saw the lightbulb in your, ugh, CPU? turn on when I said "endocrine." Keep that Zebra rifle well sighted my friend, not every hoofbeat you here is a horse ...although most are.
Although not trained to make medical diagnoses I'm impressed with ChatGPT's ability to come up with reasonable differential diagnoses. DDx is the most important intellectual process in medicine due to its complexity. Bayesian and probability weighting are often required.
Seasoned physicians can draw on years of personal experience to form probability weighted guesses on a diagnosis. ChatGPT lacks this gestalt and clearly isn't replacing me in the next 2-3 years but it wasn't made for DDx. Still, this AI is far ahead of where I thought we were.