In the film industry, we start to see how machine learning can help raise awareness about gaps in representation. Fellow Googler Hartwig Adam worked in collaboration with the Geena Davis Institute for Gender in Media, to analyze every frame from the top 100 grossing films over each of the last three years. Using a technique called classification, they compared the frequency that women and men appeared. The result: men are seen and heard nearly twice as often as women.
So where are these disconnects coming from? Are car companies intentionally trying to hurt women? Was the decision to color balance for light skin overtly racist? Are movie directors, producers, and writers trying to marginalize women on purpose? I choose another answer. I believe these were the behaviors of rational actors making what they perceived to be obvious decisions, because they were in line with defaults. Let's review one more example.
A Cornell University study performed in the late seventies—and reproduced a number of times since (1983, 1986, 2000, 2015)—asked participants to rate an infant’s emotional response after watching a video of the baby interacting with a number of different stimuli. Half the participants were told the child was a boy, and the other half that it was a girl. When the child smiled or laughed, participants agreed universally that the infant’s dominant emotion was joy. But when the child interacted with something jarring, like a buzzer or a jack-in-the-box, there was a split. If a participant had been told the child was a girl, they thought her dominant emotion was fear. But if they'd been told the child was a boy, participants thought the dominant emotion was anger. Same child, same reaction, different perception.
Imagine a teacher compiling a reading list for her students. She wants them to grasp a certain concept which is expressed in different ways in each of the books. Simply memorizing the various answers won’t result in any practical knowledge. Instead, she expects her students to discover themes and patterns on their own, so they can be applied more broadly in the future.
The majority of machine learning starts much the same way, by collecting and annotating examples of relevant real-world content (this is called “training data”). Those examples are then fed into a model that’s designed to figure out which details are most salient to the prediction task it’s been assigned. As a result of this process, predictions can be made about things the model has never seen before, and those predictions can be used to sort, filter, rank, and even generate content.
It’s easy to mistake this process for an objective or neutral pipeline. Data in, results out. But human judgment plays a role throughout. Take a moment to consider the following:
- People choose where the data comes from, and why they think the selected examples are representative.
- People define what determines success, and further, what evaluation metrics to use in measuring whether or not the model is working as intended.
- People are affected by the results.
Machine learning isn’t a pipeline, it’s a feedback loop. As technologists, we’re not immune to the effects of the very systems we’ve helped instrument. The media we get exposed to daily—including the way it’s been ranked, sorted, filtered, or generated for us—affects the decisions we’ll make, and the examples we’ll draw from, the next time we set out to build a model.
Considering the flaws in human perception, and the kinds of of less-than-ideal product decisions like those detailed in this article, one of the conclusions people often jump to is that we need to remove human judgment from the process of machine learning. “If we could just get rid of these pesky human emotions,” what a wonderfully efficient world it could be.
Let’s look at another example. In 2016, a company called Northpointe developed software to predict the likelihood that defendants would re-offend if granted parole. When all other criteria were equal, the singular difference of race dramatically boosted the risk score of blacks over whites, often by more than double. The company claimed that they didn’t use race as a feature in training their model, but unfortunately this notion of data “blindness” actually makes the problem worse. Given the nature of deep learning, and the sheer amount of information a model is exposed to, it’s all but guaranteed that proxies for race (like income or zip code, attributes that are highly correlated to race in the United States) will be automatically discovered and learned.
A particularly awkward example came from FaceApp, an app that allowed users to take a selfie and turn on a “hotness” filter. But it often just made people look whiter (it’s since been disabled). This is an example of what can happen when a model is trained with skewed data—in this case, social media behavior from predominantly white countries.
Another example caused quite a stir in the research science community early last year. Two researchers from China claimed to have trained a model that—using only a headshot photo—could predict whether someone was a criminal with almost 90 percent accuracy. There were a number of methodological issues with their approach. For example, the number of examples in their data set was rather limited, and they had no way to know for sure whether the people they labeled as ‘“non-criminals” had ever committed a crime and just never got caught. But on a deeper level, their approach assumes that there are people who are born criminals. If there are any patterns to discover in this case, they’re not about the judgement of individuals, but rather those who’re doing the judging.
There’s no silver bullet. Fairness is contextual and uniquely experienced by every individual. So when we set out to build technology for humans, using human data, we need to plan for how we’ll respond when—not if—bias manifests in some unwanted way. Removing yourself from the process and trusting in the myth of neutral technology isn’t enough, so I propose that we instead make machine learning intentionally human-centered and that we intervene for fairness.
I have many traits that would be considered defaults in today’s tech industry. I’m white. And I’m a cisgendered man, with a partner of the opposite sex. But I also have traits that are outside of the norm. I’m very tall, for one. I’m also legally blind, and as a Jew, I grew up in a cultural minority.
We’re each complex individuals—our traits don’t define us, but it would foolish to pretend others don’t see them. By taking the uncomfortable step to inventory our own traits—physical, social, cognitive, and otherwise—we can better connect with the experience of being made to feel like an outsider, when a default that doesn’t match up with our identity is invoked in day-to-day life. For many people, this kind of exercise isn’t a choice.
We all share the desire to belong. To just be ‘us’ without having to bear the burden of someone else’s preconceptions. By taking stock in our assumptions about values and goals, we can start to make room for more voices in the discussion. And hopefully in the process, we can learn to see the world as less binary: people like me, and people not like me. And instead, see it as a world of intersectionality, where we celebrate our differences as opportunities for greater understanding. And in so doing, to harness machine learning to its full potential as a tool to help us seek out patterns in the world—especially at the margins. A tool that can enrich human connection and augment human capability; a tool for explorers; to help us see beyond the defaults.
For more on how to take a human-centered approach to designing with ML and AI, visit our full collection.