Now more than ever, we need to think critically about findings published by ANY source (even McKinsey & Company) before buying in.
Example: A few days ago, the McKinsey Global Institute published a report on the gender pay gap (link below), a metric that—while far from perfect—was developed to help measure the degree to which women are often paid less than men to perform the same work, even when they bring comparable education and experience to the table.
Their report is beautifully presented, using vibrant and interactive graphs and diagrams to depict their analysis. Visually, it’s pretty convincing.
BUT - the narrative outlining their methodology/approach reveals serious issues, from overarching design missteps to puzzling analytical choices.
Here is the most glaring problem with this analysis: It cobbles together a dataset of “professional profiles” which lack the two of the most central datapoints to studying the gender pay gap issue:
Actual GENDER of the individuals whose profiles are being analyzed
Actual PAY (compensation) for those same individuals
To compensate for the missing data, the team uses a machine-learning model to infer gender; and attempts to join the de-identified “professional profiles” to salary data from the US Bureau of Labor Statistics… which is impossible to do at the individual level.
On top of this, we know accuracy can diminish with each step of a multi-stage analysis, as we layer assumptions on top of each other; this can have a measurable effect even when we’re sure of data quality and the fit between our source data and the problem being studied. But when our data isn’t fit for purpose, a complex analysis like this tends to fall apart - and that’s what we’re seeing here.
Ultimately, McKinsey's claim that we can attribute ~80% of the gender pay gap to a “work-experience” pay gap is NOT adequately supported.
It's clear their analysts have—almost quixotically—overlooked myriad data challenges and made significant leaps of logic to turn suboptimal inputs into an authoritative-sounding report.
What’s not clear is WHY. The team of authors is heavily female, and heavily credentialed. It’s hard to understand why they published a report that makes definitive and reductive claims on this topic.
McKinsey’s efforts might have been better invested in finding a better way to detect and quantify the persistent problem faced by women in the workforce: Being equally educated and qualified to their male peers, and getting paid less to perform the same job.
What’s more - this work by McKinsey’s team could be considered an interesting and creative exploration, if we frame it within better context. Here is a claim that IS supported by the data: Women whose primary goal is to increase their earning power (rather than to pursue their passion) might prioritize specific types of career moves to increase their attainment of that goal.
—
Overall, this publication by McKinsey offers us an opportunity to reflect on how to produce better analysis. How do you sanity check your assumptions and sampling methods during a large project like this? Even the best data scientists face the very real occupational hazard of going down the proverbial "rabbit hole" and arriving at faulty conclusions from a complex synthesis of multiple data sets.
Toward that end, 𝗵𝗲𝗿𝗲 𝗶𝘀 𝗮 𝗹𝗶𝘀𝘁 𝗼𝗳 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀 𝗳𝗼𝗿 𝗠𝗰𝗞𝗶𝗻𝘀𝗲𝘆, which they might have asked each other internally throughout the course of their analysis:
1. This analysis explores the impact of only two factors affecting compensation and the gender pay gap: job changes and time worked. If the datasets did not also include data for other proven contributors to the gap, how can this analysis calculate the % contribution of each?
2. The largest source of public profiles is LinkedIn. If McKinsey analysts used LinkedIn profiles, how did they account for low registration levels observed for individuals in some major employment groups - for example: High-tech, Small Business, Public Sector, Nonprofit, and others?
3. LinkedIn user engagement and data quality are highly variable. Profile information is self-reported, updated according to an individual’s preferences, needs, and job search activity. Did McKinsey investigate whether variations in engagement/data quality are uniformly distributed by gender and industry?
4. How exactly did analysts integrate de-identified data at the individual level with occupation-level data sets to determine compensation for individuals? If profile data was de-identified, what keys were available for joining? The term “integration” implies an individualized match made between each person’s profile and their salary data. That seems unlikely.
5. With its vast resources, can McKinsey not access datasets that allow comparisons of Full Time Equivalent (FTE) pay between men and women of comparable backgrounds, who performed the same jobs? Wouldn’t that data better illustrate the disparities we seek to resolve?
To read more about what others have noticed after reading through the report, check out the comments in this LinkedIn post: https://tinyurl.com/yejksm5z
Read the report for yourself at this link: https://lnkd.in/egycNah9