A separate set of patient records were used to evaluate the AI Clinician’s performance. The algorithm was used to choose a treatment, and the patients were evaluated based on whether their actual treatments were similar to the ones the algorithm recommended. Overall, the software recommended lower doses of IV fluids and higher doses of drugs that cause constriction of blood vessels. People who received similar treatments to these recommendations had the lowest mortality among this group of patients.
Diagnosis
The second paper focused on identifying problems that require treatment. The issue the focused on is bone fractures. While these are often easy to spot, small chip or hairline fractures can be difficult for even a specialist to spot. And, in most cases, the diagnosis falls to a non-specialist, typically a doctor working in emergency medicine. The new research isn’t intended to create an AI that replaces these doctors; rather, it’s intended to help them out.
The team recruited 18 orthopedic surgeons to diagnose over 135,0000 images of potential wrist fractures, and then it used that data to train their algorithm, a deep-learning convolutional neural network. The algorithm was used to highlight areas of interest to doctors who don’t specialize in orthopedics. In essence, it was helping them focus on areas that are mostly likely to contain a break.
In the past, trials like this have resulted in over-diagnosis, where doctors would recommend further tests for something that’s harmless. But in this case, the accuracy went up as false positives went down. The sensitivity (or ability) to identify fractures went from 81 percent up to 92 percent, while the specificity (or ability to make the right diagnosis) rose from 88 percent to 94 percent. Combined, these results mean that ER docs would have seen their misdiagnosis rate drop by nearly half.
Neither of these involved using the software in a context that fully reflects medically relevant circumstances. Both ER doctors and those treating sepsis (who may be one and the same) will normally have a lot of additional concerns and distractions, so it may be a challenge to integrate AI use into their process. But the success of these efforts suggests that clinical trials of AIs will be happening sooner rather than later, and then we’ll have a real sense how much they help with actual diagnosis and treatment.
Nature Medicine, 2017. DOI: 10.1038/s41591-018-0213-5 (About DOIs).
PNAS, 2017. DOI: 10.1073/pnas.1806905115 (About DOIs).