Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification

Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification is a peer-reviewed paper by Joy Buolamwini and Timnit Gebru presented at the inaugural Conference on Fairness, Accountability and Transparency (FAT*) in New York on 23 February 2018 and published in Proceedings of Machine Learning Research volume 81, pages 77–91. The work was completed while Buolamwini was a graduate student in the MIT Media Lab's Civic Media group and Gebru was at Microsoft Research.

The paper audited three commercial gender-classification APIs — IBM Watson Visual Recognition, Microsoft Cognitive Services, and Face++ (Megvii) — against the Pilot Parliaments Benchmark, a 1,270-image dataset Buolamwini assembled from the official portraits of parliamentarians in Rwanda, Senegal, South Africa, Iceland, Finland, and Sweden, balanced on gender and on the six-point Fitzpatrick skin-type scale. Its headline finding is the 43-fold gap between the maximum 0.8% error rate on lighter-skinned male faces and the up-to-34.7% error rate on darker-skinned female faces; broken out per vendor, IBM misclassified the gender of 35% of darker-skinned women, Face++ 34.5%, and Microsoft 20.8%, with error rates for the darkest-skinned women under the Fitzpatrick VI category reaching 46.5% and 46.8%. Buolamwini's signature framing of the result — "to fail on one in three, in a commercial system, on something that's been reduced to a binary classification task, you have to ask, would that have been permitted if those failure rates were in a different subgroup?" — has been carried into subsequent advocacy and policy writing on algorithmic auditing as the canonical statement of why intersectional benchmarks matter.

The paper triggered immediate vendor responses: IBM replied the same day pre-release results were shared (22 December 2017) and committed to retraining its system, with Microsoft and Face++ following in the months that followed. The 2019 follow-up audit Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products, by Inioluwa Deborah Raji and Joy Buolamwini, presented at AIES 2019, measured the resulting API revisions and documented overall Pilot Parliaments Benchmark error reductions of 5.7% for Microsoft, 7.7% for IBM, and 8.3% for Face++. A longer arc runs from the paper through the post-Floyd June 2020 corporate retreat from facial recognition — IBM's exit from the facial-recognition business and Amazon and Microsoft's police-use moratoria, each of which the firms and subsequent commentary explicitly traced back to the Gender Shades audit and the public-evidence ground it laid.

Within the corpus, Gender Shades is the seed empirical artefact of the algorithmic-accountability field on which the three existing Algorithmic Justice League-anchored Publications — Unmasking AI, Comply To Fly?, and Bug Bounties For Algorithmic Harms? — explicitly build. It fills the peer-reviewed audit-paper publication sub-type that the three AJL self-published reports do not occupy, and it is the founding output of the public voice that Unmasking AI and the "coded gaze" / "evocative audit" framings later extend. The paper is also the work AJL has continued to organise around — its project page on AJL's site hosts the original paper, the 5th-anniversary GS5-API-Results dataset, and the framing under which AJL recruits new participatory-audit cohorts — making Gender Shades not only the cluster's earliest and most-cited research output but its continuing organising anchor.