AI recognition of patient race in medical imaging: a modelling study
thelancet.comThe interpretation part hit home: "The results from our study emphasise that the ability of AI deep learning models to predict self-reported race is itself not the issue of importance. However, our finding that AI can accurately predict self-reported race, even from corrupted, cropped, and noised medical images, often when clinical experts cannot, creates an enormous risk for all model deployments in medical imaging."
"Predict self-reported race". Not race from DNA. (That's routinely available from 23andMe, and is considered an objective measurement.[1]) They should have collected both. Now they don't know what they've measured.
[1] https://www.nytimes.com/2021/02/16/opinion/23andme-ancestry-...
what's this enormous risk they're talking about? racial bias in x-ray reading? race can be a risk factor in plenty of diseases, why should we actively try to remove this information from medical images?
"This issue creates an enormous risk for all model deployments in medical imaging: if an AI model relies on its ability to detect racial identity to make medical decisions, but in doing so produced race-specific errors, clinical radiologists (who do not typically have access to racial demographic information) would not be able to tell, thereby possibly leading to errors in health-care decision processes."
Without knowing the actual outcome, isn’t there also a possibility of error due to not knowing the race of the individual? They used mammogram images in the study and it is well known that incidence of breast cancer varies by race. Removing that information from the model could result in worse performance.
Well one thing you wouldn’t want to do is take the output of this model and then apply a correction factor for race on top of it, because the model is already taking that into account.
Is that true or would it help as a tie breaker in cases where the confidence was just at or below the threshold?
Well I suppose you only care about a correction factor to a binary model when it breaks a tie. You wouldn't want to apply a tiebreaker correction twice though.
Typically? It's coded in the standard. There's a DICOM tag for it.
https://dicom.innolitics.com/ciods/procedure-log/patient/001...
Unlike the authors of this research paper I am not a trained clinician, so I can't tell you. However I would note that the first exemplary value in the link you gave me is "REMOVED".
It doesn't provide example data, but there's still a spot in the standard for it. The values can differ by modality or manufacturer. Sure, it's not required, but certainly it's very important in some situations. Consider dermoscopy.
If interested, searching for "dicom conformance" should yield lots of docs that probably contain specific values for those things.
FWIW, the standard printed out is multiple linear feet of shelf space. There is a spot for a lot of things.
One common issue is a lot of these kinds of tags rely on optional human input and are inconsistently applied. As opposed to say, modality specific parameters produced by a machine, which are consistent.
DICOM is a great example of design by committee, with the +'ve and -'ves that implies.
I don't understand that part. All modern EHRs have a field for self-reported race, and clinical radiologists do typically have access to that information. (Whether they actually look at it, or whether it's useful when reading images, are separate issues.)
ok, maybe it's an US specific thing, why wouldn't a clinical radiologist have all the information he can gather about his patient including race to help the diagnosis?
Because in the US we are required to pretend that there is no such thing as race and no such thing as gender, and all people are exactly and precisely the same and there can be no differences.
Not to get into a flame war, but I want to present an alternate option to yours.
Because in the US some people have a hard time understanding that all races and genders deserve to be treated equally as humans with the same access to goods and services. Further, that there are disparities in care based on race/ethnicity[1][2] and gender[3][4] because of that racism/sexism present in the systems. This then leads to requiring that race/ethnicity and gender data be scrubbed sometimes to keep people from impacting outcomes based on their own biases.
[1] https://www.americanbar.org/groups/crsj/publications/human_r...
[2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1924616/
[3] https://www.americashealthrankings.org/learn/reports/2019-se...
It sometimes makes sense to scrub race/ethnicity/gender information from certain types of data, typically when a human is going to be making individual decisions.
For example, not having race data on resumes is generally productive, because that categorization can't provide a meaningful input to the decision associated with an individual person. Even if it were to be the case that there was some correlation between race and skill at whatever job you're interviewing for[1], the size of the effect is almost certainly small, and in the meanwhile you've also controlled for any bias in the person doing the reviewing.
If you're having a machine look at a dataset, and the machine determines that race or ethnicity is a material factor in determining some attribute in that dataset, you're not doing anybody any good by denying that fact and destroying the result.
[1]Let's ignore for the purposes of this discussion, fields (like certain sports) where extreme competition combines with a position heavily dependent upon racially-linked physical characteristics. Though even in this case, there is still a (different, weaker) argument for suppressing race data in "resumes" (yes, I know, ballplayers don't submit resumes to their local NBA franchise)
Race is a rough, subjective, culturally-bound summary of characteristics. If you're already evaluating characteristics, adding either your guess of race or a self-reported race is like injecting gossip into good data.
If the outcome that you're trying to predict is also affected by perceptions of race, you've built a gossip feedback loop.
Then you should be looking at ethnicity and not "race" as such. For example, Ashkenazi Jews as an ethnic group are genetically very distinct from other Europeans, but are generally considered "white" on self-reported race surveys.
"Very distinct" seems a little exaggerated. Compare the "Autosomal genetic distances" between Ashkenazi jews and other European groups at https://en.wikipedia.org/wiki/Genetic_studies_on_Jews with a similar table of Intra-European distances at https://en.wikipedia.org/wiki/Fixation_index. Finns and French have like twice the distance as Italians and Ashkenazi.
Now look just above that latter table, showing distances between East Asians and Europeans. The distances are far greater--more than 10x.
The precision with which we can identify and track ancestry, often based on small fractions of DNA (Y-chromosome in particular wrt Ashkenazi Jews, not mtDNA as one might think) doesn't imply the degree of genetic distinctiveness.
>If you're having a machine look at a dataset, and the machine determines that race or ethnicity is a material factor in determining some attribute in that dataset...
I think the trickiness is in providing the machine unbiased data to begin with so that it doesn't incorrect associations between features like race. The most egregious examples I'm aware of are the machine learning systems used to suggest criminal sentencing, but, apropos to this topic I believe there are cases where it may produce erroneous associations in something like skin cancer risk.
>> Because in the US we are required to pretend that there is no such thing as race
Then you are not pretending very well. When I lived in the US I was shocked at how often it was an issue. It permeates nearly every aspect of US culture.
The icing on that cake: A government-run interactive map so you can lookup which races live in which neighborhoods. Some versions allow you to zoom in to see little dots representing clusters of black or white residents. https://www.census.gov/library/visualizations/2021/geo/demog...
Actually, the US federal government specifically recommends that healthcare providers record patients' race, ethnicity, assigned sex, and gender identity. Most of those elements are self identified.
https://www.healthit.gov/isa/uscdi-data-class/patient-demogr...
Interesting, this is like the dog learning calculus thing. We may create an AI that could perceive things that we aren't able to, or perceive things differently, because we're "limited" in a way that the AI isn't. We wouldn't be able to even tell this is going on, because we don't have the mental model in place to account for it to understand it. We'd be the dog.
> racial bias in x-ray reading?
no, it implies there is a signal in the dataset that could be something other than clinical. This means that until they can pinpoint the cause, or the thing the AI is detecting, all the other things it predicts are suspect.
ie if the AI thinks the subject is west african, then it might be more inclined to diagnose something related to sickle cell.
Or north western european woman in her mid 60s vs a japanese woman might get widly different bone density readings for the same level of "blob" (most medical imaging is divining the meaning of blobs and smears )
My first thought here is to relate this to the problem of early colour film, which was largely tested and validated with only light skin tones in mind. Once it was put out into the wild, folks with darker skin tones found the product to be total crap. Why? Because there was a glaring OOD (Out of Distribution) problem during testing.
Similarly, if the train/test sets used here - for X-ray based diagnostics - using Machine Learning relies only on specific races, then the performance might be worse for other races, given that there's a new discriminatory variable in play.
The obvious solution here is to reduce bias by ensuring race is part of the dataset used for training and testing. Which, due to PII laws in play, may actually be quite challenging! Fascinating tradeoff imo.
I don't get it either. It's accurate. It would be a problem if it got it wrong, which could, for example, underweight quantitative genetic data and adversely influence differential diagnosis.
AI is driven by the training sets, but the goal is to find the underling issues.
Suppose AI #1 got a higher score on the training data and AI #2 had a more accurate diagnosis. Obviously you want #2 but if there is bias in the training data based on race and the AI has access to race then eventually you overfit into #1.
ML models are great tools, but they're way too much of a black box. What you have here is a model that's predicting something you think it shouldn't have been possible to predict, and you can't simply ask it where that prediction comes from. Absent an explanation for how the model is doing this, you have to consider the possibility that whatever is poisoning that prediction will also poison others.
> ML models are great tools, but they're way too much of a black box.
A human doctor is also a black box, in meat form.
yep, the case for "enormous risk" hasn't been well articulated. It's been repeated a lot, but of all the problems in medical care, this isn't one of the larger ones.
What if it turns out that humans have identifiable biological differences among genetic sub-groups, ethnicities, etc? It would be anarchy in the social sciences.
soon they will want to remove race indicators for photographs and tik tok videos. who knows, maybe its racist to be of a race >.>
I suspect this is a "tank vs sky" problem. The article says that the bright areas of bone are not the most important for predicting race. What if it's some features of different hospitals and x-ray setups?
Also did they release their code and anonymized data? If not, it's impossible to tell if this is a bug.
If I got this result in my work, I would check it 10k times over because it defies belief. Even allowing subtle skeletal differences in different ethnic groups, the differences in this case are not in the bone and at least sometimes not visible to the human eye. Unless there is an undiscovered difference in radio-opacity across ethnicities, the result doesn't make sense.
Replying to my own post because I can't edit it anymore.
Apparently this is a known and persistent affect across a variety of other medical images, tests, and scans. Not just for a "race" but for ethnic groups in general, as well as biological sex. So this might actually just be an "AI hit piece" that otherwise confirms an unpalatable but persistent and strong effect in the literature. The causes seem to be badly understudied, in part due of the obvious need for delicacy and respect around such topics.
This result is tremendously implausible to me, but I am finding quite a few articles documenting similar phenomena across things like retina scans and brain MRIs.
>This result is tremendously implausible to me, but I am finding quite a few articles documenting similar phenomena across things like retina scans and brain MRIs.
As prometheus76 says, perhaps you will one of these days be able to mentally resolve the inherent contradiction in the above sentence.
What is the value of being a smug jerk, especially if you plan to be wrong?
If your prior belief points strongly in one direction, it is completely rational to require strong weight of evidence in order to update it to point to the other direction.
And yes, it's a completely reasonable prior belief for a person who is not already versed in medical imaging literature.
I often find that people who study this literature have bad attitudes like yours. You should be grateful that there are people out there who value intellectual honesty enough to acknowledge when a result is a result and to change their beliefs. Instead I get two different people showing up to insult me.
What you are experiencing is cognitive dissonance. Take your time. It's never fun.
I don't see the value in insulting people about this. I wrote a longer response here: https://news.ycombinator.com/item?id=31421346 but it applies equally well to your post.
Not too surprising that physical differences across ethnicities are literally more than skin deep. It wouldn’t be shocking that a model could identify one’s ethnicity based on, for example, a microscope image of their hair; why should bone be any different?
I’m more surprised that the distinguishing features haven’t been obvious to trained radiographers for decades. It would be cool to see a followup to this paper that identifies salient distinguishing features. Perhaps a GAN-like model could work—given the trained classifier network, train 1) a second network to generate images that when fed to the classifier, maximize the classification for a given ethnicity, and 2) a third network to discriminate real from fake X-Ray images (to avoid generating noise that happens to minimize the classifier’s loss function). I wonder if the generator would yield images with exaggerated features specific to a given ethnicity, or whether it would yield realistic but uninterpretable images.
I think it's more likely the case that (a) most radiographers aren't trained in medical school to look for distinguishing racial features (why would they be?) and (b) in most cases the radiologist knows or can easily guess the race of the patient anyway so there's no need to try to guess it from X-ray imaging data. There are a lot of anatomical features related to race that have been known since before radiology has been a field, it's just not pertinent to the job of most radiologists.
..”In this modelling study, we defined race as a social, political, and legal construct that relates to the interaction between external perceptions (ie, “how do others see me?”) and self-identification, and specifically make use of self-reported race of patients in all of our experiments.”
Garbage research.
Perfect example of citations-driven research. The authors aren’t motivated by a genuinely interesting scientific question (“are anatomical differences between genetically distinct groups of people visible in X-rays?”). Instead, the authors know that training a classifier to predict race will generate controversial headlines and tweets. All publicity, positive or negative, leads to more citations.
> genetically distinct groups of people
Is race a genetically distinct marker though? I guess if you limit the sample enough it is, but I've always thought of race as more of a continuous quality than a distinct one.
Race _is_ a spectrum but genetic differences themselves are distinct (SNPs). It's trivial to train a classifier to distinguish race from genetic data, hence, I'd argue they are distinct groups.
You can draw an analogy to colours in the rainbow: a rainbow is a spectrum but we can still draw lines that demarcate colours. Colour definitions are fuzzy at the edges but this doesn't mean coarse colour labels are not distinct.
If we look at many many other mammals it is very clear that there is breeds. And those breeds are noticeably different, just think of cats, dogs, cows, pigs and so on. I really see no reason why this wouldn't extend to humans as well. And that the distinct groups wouldn't have some markers. Like longer than average bones.
Now, issue really is that whole race grouping is extremely murky. And not really anywhere specific enough as used in common speech. White, Black, Asian etc. are way too wide to be very useful. Even inside what we could understand as rather homogenous groups there is lot of difference between areas.
If it's trivial, why are results so variable on different ethnicity websites? And what happens when the definition of "white" changes in 30 years to unambiguously include Latinos?
(Hint : what you want is a classifier on ethnicity, and those aren't trivial either)
There’s nothing trivial about determining race in healthcare research.
Feel free to cite sources.
This is the only reasonable possible way to do it. Races are fluid and ill-defined constructs, so self-identification is the best you can do for ground truth.
And you are qualified to make that assessment because...?
The submitted title ("AI identifies race from xray, researchers don't know how") broke the site guidelines by editorializing. Submitters: please don't do that - it eventually causes your account to lose submission privileges.
From the guidelines (https://news.ycombinator.com/newsguidelines.html):
"Please use the original title, unless it is misleading or linkbait; don't editorialize."
It's the title of the Vice article about the same topic. https://www.vice.com/en/article/wx5ypb/ai-can-guess-your-rac... (It was posted last year.) (No idea why the OP used one title and another URL.) (The title of Vice is a bad title anyway.)
This or similar is the title on multiple sites.
https://www.boston.com/news/health/2022/05/18/scientists-cre...
https://nationalpost.com/health/health-and-wellness/ai-can-t...
https://www.sciencealert.com/ai-can-predict-people-s-race-fr...
https://www.iflscience.com/technology/ai-can-identify-race-f...
https://www.bostonglobe.com/2022/05/13/business/mit-harvard-...
Just a small collection
Good catch!
The fact that the model seems to be able to make highly accurate predictions even on the images in Figure 2 (including HPF 50 and LPF 10) makes me skeptical. It feels much more probable that this is a sign of data leakage than that the underlying true signal is so strong that it persists even under these transformations.
https://arxiv.org/pdf/2011.06496.pdf
Compare the performance under high pass and low pass filters in this paper on CIFAR-10. Is it really the case that differentiating cats from airplanes is so much more fragile than predicting race from chest x-rays?
> Models trained on low-pass filtered images maintained high performance even for highly degraded images. More strikingly, models that were trained on high-pass filtered images maintained performance well beyond the point that the degraded images contained no recognisable structures; to the human coauthors and radiologists it was not clear that the image was an x-ray at all.
What voodoo have they unearthed?
I tend to not believe unbelievable results in machine learning. It's too easy to unintenionally cause some kind of information leakage. I haven't read the paper in detail though, so their experimentation setup could be foolproof, this is not a critique of this paper specifically.
This reminds me of the ML research that could predict sex from an iris. It turns out they were using entire photos of eyes to do this. There are so many obvious cues to pick up on in that case, like eyeliner, eyelashes being uniform (or fake), trimmed eyebrows, general makeup on the skin, etc.
> I tend to not believe unbelievable results
That seems tautologically true.
> What voodoo have they unearthed?
Curious for the take not of a neuro-ophthalmologist. If they too are stumped, this may be a path to a deeper understanding our visual system.
Simple transformations obviously discernible to us blind computer vision. (CAPTCHAs.) There may be analogs for human vision which don’t present in the natural world. Evidence of such artefacts would partially validate our current path for artificial intelligence, as it suggests the aforementioned failures of our primitive AIs have analogs in our own.
I think there's a significantly greater than zero chance that they simply botched their ML pipeline horribly and would get their 0.98 AUCs from completely blank images.
I think it’s pretty straightforward. Imagine the fourier transforms of some recognizeable audio signals. Maybe a symphony and a traffic jam. They’ll look totally different, even to the naked eye. If you chop off the low frequency components, you can still probably tell which fourier spectrum is which. But now do the same thing in time domain (high-pass filter the audio). It probably won’t be clear that you’re listening to a symphony anymore.
... Adversarial examples.
It's a whole field of research, and it's pretty trivial to generate them for most classes of ML models. It's actually quite difficult to create robust models that DON'T have this problem...
If you’re interested in “hard to describe features that can be learned with enough expiration”, look up chick sexing
Interesting field. You have to breed a couple of males to maintain the species. If you were to pick those from the mis-sexed group I suppose natural selection would reduce the classifying feature over time. I wonder if poultry farms pick a couple of the male-classified birds to maintain a stock of well identifiable males and kill all the mis-classified males.
It would be nice to see more genuine, enthusiastic scientific curiosity to understand how the ML algorithms are doing this, rather than just abject terror and alarm.
It seems like the reason the researchers in this paper are concerned is precisely that they tried and failed to understand how the ML algorithms are doing this. If they’d discovered that white people have a subtly distinctive vertebra shape the model was detecting, it would have been much more of “oh, we discovered a neat fact”.
I don't think they tried very hard at all. I see no meaningful use of modern explanation tools.
There are lots of known ways in which people of different races are different physiologically. Probably even more unknown ways.
There could also be differences in imaging technology used in different communities, as others have suggested. I'd be a bit surprised if something like that could create such a strong signal but it's on the table.
For those of us less familiar with this space, what are these modern explanation tools? (I certainly agree that it's plausible the model is seeing a physiological difference, and the researchers seem to have considered a few concrete hypotheses on that dimension.)
Here's an introduction to one technique: https://cloud.google.com/blog/products/ai-machine-learning/e...
This is a cutting edge subfield of ML, so it's understandable that one paper in a medical journal isn't going to be on that cutting edge, but I think they should at least acknowledge that their investigations barely scratched the surface.
Thanks! This looks pretty neat, I'll have to dig more into their explainable AI product later.
"This issue creates an enormous risk for all model deployments in medical imaging: if an AI model relies on its ability to detect racial identity to make medical decisions, but in doing so produced race-specific errors, clinical radiologists would not be able to tell, thereby possibly leading to errors in health-care decision processes."
Why would a model rely on its ability to detect racial identity to make decisions?
What kind of errors are race-specific?
Let's say you're trying to train an model to predict if a patient has a cancerous tumor based on some imaging data. You have a data set for this that includes images from people with tumors and people without, from all races. However, unbeknownst to you, most of the images from people of race X had tumors and most of the images from people of race Y did not have tumors.
If the AI is also implicitly learning to detect race from the images, it's going to learn an association that people of race X usually have tumors and people of race Y usually do not.
The problem here is that the people training the model and the clinical radiologists interpreting data from the model may not realize that race was a confounding factor in training, so they'll be unaware that the model may make racial inferences in the real world data.
If people of race X really do have a higher incidence rate for a specific type of cancer than race Y, maybe this is OK. But if the issue is that there was bias in the training/validation data set that was unknown to the people building the model, and in the real world people of race X and race Y have exactly the same incidence rate for this type of cancer, then this is going to be a problem because it's likely to introduce race-specific errors.
Just because the model relies on race in some way doesn’t mean that we know it relies on it. I.e., the model is, unbeknownst to us, biased on race in inaccurate ways.
Presumably the model would actually be biased on race in accurate ways, if it found the correlation itself
I could be entirely wrong here, so if you've got more context in this area by all means correct me.
Consider an "AI" that rates the probability of recidivism for prisoners nearing their parole date. That score would then be presented to the parole board, and taken into consideration in determining whether or not to grant parole. If this AI were accidentally/incidentally accurately determining the race of the prisoner, then the output score would take that into account as well. Black men have a recidivism rate significantly higher than other groups[1]. The reasons for the above aside - it's a complex topic, and outside the scope of this analogy - this is extremely undesirable behavior for a process that is intended to remove human biases.
You might then ask, how does this relate to medical imaging? Medical decisions are regularly made based on the expected lifespan of the individual. It makes little sense to aggressively treat leukemia in a patient who is currently undergoing unrelated failure of multiple organs. Similarly it would likely make sense for a healthy 30-year-old to undergo a joint replacement and associated physical therapy, because that person can reasonably be expected to live for an additional 40 years while the same treatment wouldn't make sense for a 70-year-old with long-term chronic issues. This concept is commonly represented as "QALY" - "quality-adjusted life years".
Life expectancy can vary significantly based on race[2].
An AI that evaluates medical imagery that considers QALY in providing a care recommendation may result in a positive indicator for a white hispanic woman and a negative indicator for a black non-hispanic man, with all else being equal and with race as the only differentiator.
In short - it's not necessarily a bad thing for a model to be able to predict the race of the input imagery. The problem is that we don't know why it can do so. Unless we know that, we can't trust that the output is actually measuring what we intend it to be measuring.
1: https://prisoninsight.com/recidivism-the-ultimate-guide/ 2: https://www.cdc.gov/nchs/products/databriefs/db244.htm
At the risk of discussing sensitive topics on a platform ill-suited:
If, in your hypothetical recidivism case, an AI "accurately" determined that a pattern of higher recidivism-related features was correlated to race, and was able to determine "accurately" that the specific subset of recidivism-related features predicted race, why would it be wrong to make parole decisions using those recidivism-related features?
Because both the original conviction and any recidivism is determined through the decision-making of people who are aware of race and racial stereotypes. The AI would just be laundering the decisions you were already making, not improving them.
edit: imagine I was a teacher who systematically scored people with certain physical characteristics 10% lower than people who didn't have them. Let's say, for example, that I was a stand-up comedy teacher that wasn't amused by women.
If I used an AI trained on that data to choose future admissions (assuming plentiful applicants), I would end up with an all-male class. If this happened throughout the industry (especially noting that the all-male enrollment that I have would supply the teachers of the future), stand-up comedy would simply become a thing that women were seen as not having the aptitude to do, although nobody explicitly ever meant to sabotage women, just to direct them into something that they would have a better chance to succeed in.
If you decided on race, in this instance, you would be making people much more deterministic as a result of the power of race. Race is too broad a concept to reliably say that all white people are at X chance of recidivism. Instead we want to know if Marlowe is at risk of high recidivism based on her character.
Both responses address the problem with a human making a biased decision based on race, which I think mostly we all agree would be bad.
The question I was posing is different, though, because this was discussing an AI system that looked at the underlying [in this case, recidivism] data which had race and race-adjacent information removed, and the AI has effectively rediscovered the concept of "race" by connecting it to some set of attributes of the actual [in this case, recidivism-predicting] features. If the AI were to determine such a link, that doesn't make its results biased, it just makes them uncomfortable. It's not clear to me that in such a case that would mean that we should remove those [recidivism-predicting] features from the dataset just because they ended up being correlated to race.
Maybe, maybe not. Hard to say—which is the problem they call out in the paper
> efforts to control [model race-prediction] when it is undesirable will be challenging and demand further study
The correlation being "undesirable" to the individuals doing the research does not mean that the correlation is inaccurate.
I mean, sure, there are tons of ways for garbage data to sneak into ML models -- though these guys tried pretty hard to control for that -- but if the model actually determined that "race" is a meaningful feature, then that might be because it is, and science should be concerned with what is, not with what we wish were.
If one believes and proclaims that they have controlled for variable X, but they haven’t actually done so, then their results and analysis may well be invalid or misleading because of that. Whether they actually should have controlled for X or not is orthogonal.
Oh, yes, sorry. If by the correlation being possibly-undesirable you meant that it was possibly-spurious due to incompletely controlling for some bias in the source data, then yes, conclusions based on a model which found such a spurious correlation caused by incomplete input control might be undesirably biased in a not-accurate fashion.
This study appears to have done a good job controlling for known biases that could have been proxies for race, but it is presumably possible that they missed something and tainted the data
Right, and that’s pretty much the conclusion: our explicit goal was to control for race, and yet, we appear to have failed and don’t know why (so don’t know how to adjust the control yet). So likely others using similar-enough methodologies and techniques are unknowingly failing to control.
Using race as an independent factor to make medical decisions isn’t unheard of today. The medical community is largely trying to stop doing that as a matter of social policy, so it’s a problem for that goal if an AI model might be doing it under the hood.
See e.g. https://www.ucsf.edu/news/2021/09/421466/new-kidney-function...
It could actually be the skin, it's designed to block rays, it might also have a different x-ray opacity, and that can be judged from the whole picture in particular where there's several layers of melanin, or there's transitions from melanin to very little like on hands and feet. Eyelids too, if they're retracted. And at the perimeter, the profile, different angle for the ray.
And the intention is for melanin to block x-rays too, block all rays, not just UV but deeper. Well it has a spectrum, that cannot be denied. And if you're taking all the pixels in an image, there might be aggregate effects as I described. You get a few million pixels, let AI use every part of the buffalo of the information of the picture, and you can get skin color through x-rays.
The question is what this says about Africans with light-skin strictly because of albinism, ie lack of pigmentation, but otherwise totally African.
What does this mean in terms of race being a social construct/concept?
Ever more complicated attempts to bridge the gap by muddying the waters.
Frankly, even a freshly arrived alien from Mars or Titan could easily tell Icelanders, Mongols and Xhosa apart, without knowing anything about our culture. The fact that there has been a lot of interbreeding/admixture since the Age of Sail began, does not mean that there aren't meaningful biological differences between the original groups, which still obviously exist.
An analogy: much like the existence of twilight does not render the concept of night and day a 'social construct' either. We attach certain social meanings to those natural phenomena, and a 'working day' can easily stretch into 'astronomical night' (all too often!), but that does not mean that 'night' and 'day' do not exist outside of our cultural reference framework.
There is a social concept of 'race' which corresponds to the 'working day' concept in this analogy, e.g. 'BIPOC', claiming Asians as 'white adjacent' or classifying North Africans or Jews as 'white', even though they may not necessarily look white. But this is almost certainly not what the AI identified. This social concept of race would confuse a Martian alien unless he started to study the social and racial history of the U.S., and possibly even afterwards. It definitely confuses me, a random observer from Central Europe.
One of the major issues with biological race realism is that common race classifications do not form monophyletic groupings. If you already have a set of definitions you need to fit, any sufficiently large set of features can trivially classify those categories. It doesn't matter if the groups are race or "people who like licorice". When the features and population sizes aren't uniformly distributed, you can even get very strong correlations, without there being a meaningful underlying basis.
The social definition is used because that's a most scientifically meaningful and useful definition that avoids many of the issues with biological race realism.
I've noticed people in all parts of the political spectrum have a hard time understanding the term "social construct". It doesn't mean the same as "completely made up".
Nations are uncontroversially recognized as a social constructs. However I'm certain that AI could also detect images taken outdoors in Mexico vs those in Finland. Additionally I, a US citizen, cannot simply declare that I am now a citizen of France and expect to get a French passport.
However it also means that what a nation is, is not set in stone for eternity. It means that different people can debate about the precise definitions of about what defines a particular nation. It means that Czechoslovakia can become the Czech republic and Slovakia. It means that not everyone agrees if Transnistria is an independent nation. It means that the EU can decide that a German citizen can have the same passport as a French citizen.
As a more controversial example, this is also the case when people talk about gender being a "social construct". It doesn't mean that we can simply pretend like the ideas "men" and "women" doesn't exist (as people both declare and fear). But it does mean there is some flexibility in these terms and we as a society can choose how we want these ideas to evolve.
Society is a complex and powerful part of our reality, arguably more impactful on us from day to day than most of physics (after all we did survive for hundreds of thousands of years without even understanding the basics of physics). Therefore something being classified as a "social construct" doesn't mean it "isn't real". Even more important is that individuals cannot choose who social construct evolve. I cannot, for example, declare that since taxes are a social construct, I'm not paying them anymore. We can however, as a society, change what and how these constructs are interpreted.
Still is? The AI is correlating biological features with self reported race. There are biological differences between people who have different ancestors. Finns are different from brits. The spanish are different from russians. Nigerians look different than somalians. The Japanese look differnet than filipinos.
Race picks specific and arbitrary differences , for example hispanic is a different race in US society but black and white based on skin color are as well, indians and east asians are also one "race".
Ethnicities are not social constructs but race is. The AI finds ethnic differences and correlates them with self-percievied social/racial classification.
"Race" as the evil social construct it is, takes ethnic differences and intrprets them to mean some ethnicities are different races of humans than others as in not just different ancestors but differently created or evolved despite all evidence and major religion saying all humans are one species (homosapiens) that have a common homosapien ancestor.
I thought all this was obvious but the social climate recently is very weird.
I believe you’re inverting race and ethnicity.
From national geographic: “Race” is usually associated with biology and linked with physical characteristics such as skin color or hair texture. “Ethnicity” is linked with cultural expression and identification.
I am not. Historically ethnicities and ancestoral lines aligned so ethnic differences and biological differences due to generational mating choices largely influenced by the culture that is a component of the ethnicity are aligned as well. Race is not a biological classification because it is appearance based but arbitrary. Appearance is not the same as biology. A husky appears similar to a small wolf but it might be correct to consider them (dogs) a race of wolves.
The deceptively evil part of the concept of race is, it does not simply differentiate biological features but it goes on to impose a fork at the root of the ancestoral tree where people of that race share the same origin and same differences. In reality biological differences are a result if what a culture considers attractive multiplied by mutations that help people adopt to different environments (e.g.: skin color being a result of adaptation to sun light and vitamin d levels instead of a being a feature that shows ancestoral forks in creation or evolution).
It is simply inaccurate to label people by race but it is useful to impose social evils. But biological differences due to mating and cultural choices are very real and can be examined at a granular level that takes the actual factors for the differences into account instead of the lazy+evil correlation that is the concept of race.
Ethnicity is not what culture you identify with. You don't become ethnically african american because you like african american culture and grew in a specific neighborhood. It is the marriage of culture and ancestry.
Medical software used in the US classifies Hispanic as an ethnic group, not a race. Those are separate fields in a patient's chart. Here is the official federal government guideline.
https://www.healthit.gov/isa/taxonomy/term/741/uscdi-v2
https://www.healthit.gov/isa/taxonomy/term/746/uscdi-v2
(I'm not claiming that this is an optimal approach, just pointing out how it works in most software today.)
Yes, but socially when you ask people their race they will say black or hispanic or white. And with little consitency. It is more of a way to justify and impose social classes.
So isn’t the “evil social construct” part actually the invalid extension of the theory that biological or phenotypic differences mean that someone is more or less human? You can remove that part and still acknowledge that there are biological differences between people based on their genetic lineage without invalidating their basic humanity.
The evil part is not differences but considering people as part of a different race of humans because of those differences.
Race as an entire concept to me has always been stupid at best. Sure, there are vast swaths of biological similarities (typically, though not necessarily) according to general geographic regions of the globe, but the real mistake was trying to give this vague concept a label. Can anybody give a definition of what "white" or "black" REALLY means? It's an impossible task. If we're talking just visually about skin color, congratulations, that represents citizens of some 100+ odd countries on the planet and just as many (if not many more!) cultures and languages. But leave it to humans to try and over optimize and try to denote evermore meaningless abstractions...
Let the social "culture war" rage on. The only war I see going on in the west (U.S. mostly) is a _lack_ of culture.
To me it is the epitome of laziness. The ultimate expression of the banality of evil.
Similar to prejudice and stereotyping or the worst of lies there is some truth in its reasoning but the untrue part, the lazy part allows people to commit evil and be unjust. A reason to harm others with minimal conscious discomfort.
Race, in terms of physiology has never been regarded by science to be a social construct.
In fact it can be medically harmful to think this way.
Unfortunately, that's not quite true. Here's the AMA[1] with a press release entitled "New AMA policies recognize race as a social, not biological, construct".
They discourage using race as a source of any physiological signal. They do allow using genetics, but the relevant situations are the many many ones where genetic testing isn't possible or doesn't yet provide useful signal.
Unaccountable institutions get captured very easily, and the race cult that's swept through our educated class has been a very powerful one.
[1] https://www.ama-assn.org/press-center/press-releases/new-ama...
One of the reasons certain communities were hit harder with Covid was vit D deficiency as a consequence of skin color.
That is one hypothesised cause for the disparity, social factors in those cases need to be controlled for.
A better discussion is around sickle cell anaemia[0] which is exclusively carried by people of African or Afro-Caribbean descent.
That's a better example, thank you. Reminded me that I know quite a few people with Thalassemia/Mediterranean anemia.
Sickle cell disease is exclusively caused by genetics, not race. The vast majority of people of African or Afro-Caribbean descent aren't carriers, so have the same likelihood as everyone else who is not a carrier to develop it.
Skin color isn't race.
There is no scientific, consistent way to define race. The groups we put people into is fairly arbitrary. They don't correlate to appearance, genetics, country of origin, etc.
An interesting question in the U.S. is "who is considered white?" There was a Supreme Court case in which someone who was literally from the Caucasus was ruled not white. This is why it's sociological, not scientific.
https://www.sceneonradio.org/episode-40-citizen-thind-seeing...
Alloco 2007 looked at random locations of single nucleotide polymorphisms(SNPs) and found that, using random SNPs, you still get very good correspondence between self-identification and best fit genetic cluster. Using as few as 100 randomly selected SNPs, they found a roughly 97% correspondence between self-reported ancestry and best-fit genetic cluster.
Were the formulations of genetic clusters created through marking samples with self-reported race? If so, why couldn't you create an entirely different rubric of race by choosing a few arbitrary features to define each of them and find exactly the same thing?
If there's no scientific, consistent way to define race, how is it that a machine learning model is able to pick the race that somebody self-identifies as consistently? The model is simply using rules based on math to deduce an accurate guess.
It still is. Just because it includes physical signifiers that can be measured doesn't mean it isn't still a social construct.
To give a contrived example; if I say people with ring fingers over 3 inches long are Longfings and people wkth ring fingers 3 inches or less are Shortfings, and then out society treats people differently based on being Longfing or Shortfing, this is a social construct that is causing problems for people based on a contrived criteria that has no real meaning. The same is true of race.
What if shortfings tend to be drastically taller, and the longfings are complaining that they're overrepresented in jumpball?
> What if shortfings tend to be drastically taller
What does it mean for shortfings to be dramatically taller? Are you saying that shortfings must transmit height along with finger length; some sort of race invariance? Or are you saying that most shortfings you meet are also tall?
If a black person is a pale as a white person, they're still considered black (and may share many other characteristics that many black people have.) If some of your shortfings have long fingers, does the distinction still make sense as a scientific category?
> the longfings are complaining that they're overrepresented in jumpball?
Is admission to jumpball determined by finger measuring, or through social factors?
Perhaps there is some quality of the x rays themselves that is different? Maybe white people tend to visit hospitals with newer, better equipment or better trained radiographers and the model is picking up on differences in the exposures from that.
From the paper "Race prediction performance was also robust across models trained on single equipment and single hospital location on the chest x-ray and mammogram datasets"
They mostly accounted for this: >Race prediction performance was also robust across models trained on single equipment and single hospital location on the chest x-ray and mammogram datasets
Sure, it’s possible that bias due to the radiographer is the culprit, but this seems unlikely.
Interesting hypothesis but I can't tell if you're being serious
Same as it always did, as humans have long claimed to be able to distinguish race simply by looking.
I'm just going to abandon the term race because nothing constructive is going to come from it. It is not contentious that there are various physiological developments among groups of humans.
> various physiological developments among groups
This is a very contrived way to say that people share characteristics with other people. The real question is why people don't say that I belong to the six-foot tall bad-knees race.
Really? It's very contrived?
I'm not here to tell you what to do. Use race then. I offered up why I think this article is only generating interest is because race is a loaded word, and if it weren't used, it'd be passed over.
> The real question is why people don't say that I belong to the six-foot tall bad-knees race
This is an article about ML accurately predicting self-identified race. This is not even on the spectrum of real questions.
What term are you going to use instead? Subspecies? Breed?
I don't know. My hunch is that these suggestions, though, will be received poorly.
So your solution to people arguing in bad faith is to be so wordy that they give up and move onto more easily mischaracterized targets?
It means that is a lie.
Science does not claim that race is a social construct/concept...
While I agree with you that “social construct” isn’t the right way to think about it, the authors in this very paper say that it is.
Simply go to google image and search: "skeletal racial differences".
subspecies are found across species-- they happen based on geographic dispersion and geographic isolation, which humans underwent for tens and hundreds of thousands of years.
Welcome to the sciences of anatomy, anthropology, and forensics.
other differences:
- slow twitch vs fast twitch muscle
- teeth shape
- shapes and colors of various parts
- genetic susceptibility to & advantages against specific diseases
Just like Darwin's finches of the Gallapogos, humans faced geographic dispersion resulting in genetic, diet (e.g. hunter-gatherer vs farmer & malnutrition), and geographical (e.g. altitude) differences which over the course of millennia affect anatomical differences. We can see this effect across all biota: bacteria, plants, animals, and yes, humans.
help keep politics out of science.
> skeletal racial differences
£10 says that its not that. Anatomy is extraordinarily hard, and AI isn't that good, yet. Sure different races have different layouts, but often that's only really obvious post mortem. (ie when you can yank out the bones and look at them, there are of course corner cases where high res CAT/MRI scans can pull out decent skeletal imagery in 3D) There are other cases, but that should be easy to account for.
If I had to bet, and I knew where the data was coming from, I'd say its probably picking up on the style of imaging, rather than anything anatomical. Not all x-rays have bones in, and not all bones differ reliably to detect race.
> keep politics out of science.
Yes, precisely, which is why the experiment needs to be reproduced, and theories tested through experimentation. The reason why this is important is because unless we workout where this trait is coming from, we cannot be sure the diagnosis is correct. For example those with sickle cells have a higher risk of bone damage[1] which could indicate they are x-rayed more. This could warp the dataset, causing false positives for sickle cell style bone damage.
[1]https://www.hopkinsmedicine.org/health/conditions-and-diseas...
>If I had to bet, and I knew where the data was coming from, I'd say its probably picking up on the style of imaging, rather than anything anatomical. Not all x-rays have bones in, and not all bones differ reliably to detect race.
This was my guess as well. I've spent a lot of time around radiology and AI (I used to work at a company specializing in it) and we read a lot of the failure cases as well. There was one example where the model picked up on the hospital, and one hospital was for higher risk patients- so it learned to assign all patients from that hospital to the disease category simply because they were at that hospital.
There are a ton of cases like this out there, especially when using public datasets (which in the medical field tend to be very unbalanced datasets due to the difficulties of building a HIPAA compliant public dataset).
> one hospital was for higher risk patients- so it learned to assign all patients from that hospital to the disease category simply because they were at that hospital.
That just sounds like poor feature selection/engineering. Garbage in, garbage out.
Yeah there are definitely ways they would have avoided that, but it's just one example of many. The whole point of ML is that it picks up on learned patterns. The problem is that it can be difficult to identify what it is learning from- this paper itself says they do not know what is causing it to make these predictions. As a result it is difficult to validate that the model is doing what people think it is.
>I'd say its probably picking up on the style of imaging, rather than anything anatomical
Certainly possible! They do control for hospital and machine …
>Race prediction performance was also robust across models trained on single equipment and single hospital location on the chest x-ray and mammogram datasets
… but it’s also possible that different chest x-rays were being used for different diagnostic purposes and thus have a different imaging style, which a) may correlate with ethnicity and b) does not appear to be explicitly controlled for.
The weak factor in these AI studies seems to be data set normalization.
I wonder if some communities use certain x-ray machines verses which machines are commonly used by other communities and this has nothing to do with race but the machine being used. I read over the paper but didn't really understand it. Maybe all this is doing is identifying which machine was used.
The article is pretty fascinating and I recommend that you actually read it. For example:
>"We found that deep learning models effectively predicted patient race even when the bone density information was removed for both MXR (AUC value for Black patients: 0·960 [CI 0·958–0·963]) and CXP (AUC value for Black patients: 0·945 [CI 0·94–0·949]) datasets. The average pixel thresholds for different tissues did not produce any usable signal to detect race (AUC 0·5). These findings suggest that race information was not localised within the brightest pixels within the image (eg, in the bone)."
One of the primary ways of identifying possible race from bones in anthropology involved calculating ratio from lengths. Good for an estimate or fallback, but not completely accurate. Removing the density would do absolutely nothing to obscure that method. Any image will allow you to measure ratio of bones sizes.
So just from the silhouette of a skeleton, if I understand that correctly?
Even after being munged into a nearly-uniform gray by high pass the effect seems pretty robust.
The problem with 'race' as a concept isn't that you can genetically tell people apart.
Our tools are so precise you can tell which parent a set of cousins had with DNA tests, this doesn't make them a different species/sub-species or race from each other, even if one group has red hair and the other has black.
It's the pointless lumping together of people who are genetically distinct and drawing arbitrary, unscientific lines that's the issue.
Presumably the same experiments that can detect Asian Vs Black Vs White could also detect the entirely made up 'races' of Asian orBlack, AsianorWhite and WhiteorBlack since those are logically equivalent.
So are the races I made up a moment ago real things? No. But a computer can predict which category I'd assign, doesn't that make them real and important racial classifications? No it means my made up classifications map to other real genetic concepts at a lower level, like red hair.
Then problem is that human experts sometimes can't tell the difference while the model can.
AI is also able to determine your sex from your retinal scan with very good levels of certainty (provided that your retina is healthy; its ability to tell sexes apart drops in diseased retinas). [0]
Which came as a surprise to the ophthalmologists, because they aren't aware of any significant differences between male and female retinas.
[0] https://www.researchgate.net/publication/351558516_Predictin...
I am surprised that this is a surprise. At least color vision is encoded in the X-Chromosome so there should be variation as males have only one which can be expressed.
It is a surprise, because the retina as an organ is very well visible and observable in living people, so we have a ton of observational data and practical clinical experience. But despite that, humans haven't noticed anything.
I've never seen a retina which had been separated from a human.
One idea is that there is some difference in the x-rays themselves that could potentially be explained by racial disparities in access to (and quality of) healthcare. Maybe white people tend to visit hospitals with newer, better equipment or better trained radiographers and the model is picking up on differences in the exposures from that.
> We also showed that the ability of deep models to predict race was generalised across different clinical environments, medical imaging modalities, and patient populations, suggesting that these models do not rely on local idiosyncratic differences in how imaging studies are conducted for patients with different racial identities.
They mostly accounted for this:
>Race prediction performance was also robust across models trained on single equipment and single hospital location on the chest x-ray and mammogram datasets
Sure, it’s possible that bias due to the radiographer is the culprit, but this seems unlikely.
That's an interesting confounding variable. I think it's disproven by the fact that the AUC is too high given your hypothesis.
These results seem too accurate to be explained only by a correlation to the medical equipment used.
I mean, if color of skin, form of eyes and other visible, "mechanical" characteristics can be different it's not that big of a leap to observe that certain non-visible characteristics can differ too between humans.
Physiologies are created by genetics, and differences in ancestry are the basis for self-identified race.
Ordinary computer vision can also identify race fairly accurately, the high pass filter thing is merely pointing out that ML classifiers don't work like human retinas.
It's astonishing how many epicycles HN comments are trying to introduce into a finding that anyone would have predicted. Research which confirms predictable things is valuable of course, but no apple carts have been upset.
I would guess a causal chain through environmental factors, given how much archeologists are able to tell about prehisotric humans’ lives based on bone samples.
Bone density, micro fractures and deviations in shape. The mongols had famously had bowed legs from spending a majority of their waking lives on horseback.
I recall seeing a paper in the early 2010s with an algorithm that could discriminate between white and Asian based on head MRI images. I'm having trouble finding it now, but this finding to me is not too surprising.
So there’s material differences that supports certain prejudices; big surprise, turns out human societies have been (and still is) working very hard for thousands of years to craft those differences - isolating, separating, enslaving, oppressing, exiling their scapegoat “others”. The question is not whether the differences are real, but whether we can prevent AI from being used to perpetuate those differences. TBH, we don’t stand a chance; we live in a society where most people cannot even wrap their heads around why it shouldn’t perpetuate those differences.
> Importantly, if used, such models would lead to more patients who are Black and female being *incorrectly* identified as healthy
I think this is the point a lot of people are missing; they think, "So what if 'black' correlates to unhealthy and the model notices? It's just seeing the truth!"
However, I'm still wondering how this incorrectness works; can anyone explain?
Edit: Clue: The AI is predicting self-reported race, and the authors indicated that self-reported race correlates poorly to actual genetic differences.
My guess is that they are using an american dataset. This I would suspect encodes socioeconomic data into the samples. ie rich people, have access to better diagnostics, get seen earlier and are treated sooner. Conversely poorer present later and with more obvious symptoms. also the type of system used to take the images would also be strongly correlated.
If this is true I suspect a human could be trained the same way.
I read once that a radiologist can't always explain what they see in an image that leads them to one diagnosis or another, they say that after seeing many of them they just know.
So I suspect the same could be done for race. This would be a super interesting thing to try with some college students - pay them to train for a few days on images and see how they do.
Given the complexity of datasets, and what is known about the quality of medical scanners, is it possible that underserved communities (ie higher noise scanners) serve a specific community that is heavily skewed in race distributions?
"our finding that AI can accurately predict self-reported race, even from corrupted, cropped, and noised medical images"
It doesn't seem like noise in the images is a factor
Imaging artifacts may persist despite corruption
A lot of people are proposing simple reasons why this could be the case. They did so last year when the study that inspired this got published.
Maybe this needs to be updated from physicists: https://xkcd.com/793/
I don't trust medical journals or experimental AI research to be particularly scientific so I'll just throw this into the meaningless bin for now.