Discriminating Systems: Gender, Race, and Power in AI

Discriminating Systems: Gender, Race, and Power in AI is a report published by the AI Now Institute on 1 April 2019, authored by Sarah Myers West, Meredith Whittaker, and Kate Crawford. The report's central argument is that the AI industry's workforce diversity crisis and the discriminatory behavior of AI systems are two sides of the same problem — not parallel phenomena suited to separate HR and technical fixes — with a homogeneous workforce producing systems that encode its creators' assumptions, and those systems then shaping the conditions under which future researchers are hired, retained, and heard. The report synthesized existing literature on gender, race, and class in technology workplaces and in deployed AI systems, producing twelve concrete recommendations directed at companies and academic institutions. It is the first dedicated AI Now Institute publication — distinct from the Institute's annual multi-domain AI Now Reports — to take the workforce-to-system discrimination nexus as its whole unit of analysis, framing AI bias not as a technical artifact to be corrected downstream but as the output of discriminatory industries that extend their patterns forward into every system they ship. Its release in April 2019 came five months after the November 2018 Google Walkout — in which 20,000 Google employees walked out over a documented culture of inequity and sexual harassment — drawing on that same institutional moment to position its diversity statistics not as abstract workforce facts but as evidence of a structural condition with direct consequences for what gets built.

Workforce diversity: the numbers

The report assembled the available diversity data across the AI research and industry ecosystem into a single picture. Only 18 percent of authors at leading AI conferences are women, and more than 80 percent of AI professors are men. Within industry, women comprise just 15 percent of AI research staff at Facebook and 10 percent at Google. On racial representation, Black employees make up 2.5 percent of Google's workforce and 4 percent each at Facebook and Microsoft; at NeurIPS 2016, only six Black attendees were counted among 8,500 participants. The historical arc is not one of progress: women's share of computing jobs fell to 26 percent by 2013, below where it had stood in the 1960s, and nearly half of women in tech eventually leave the field. The report also documented how companies systematically manipulate reported diversity data — limiting pay-equity analyses to job categories with more than 30 employees (which excludes executive roles where male overrepresentation is most pronounced), covering only 80 percent of their workforces in diversity disclosures, using binary gender attribution that renders transgender and non-binary workers invisible, and excluding equity and bonus compensation from pay-equity calculations — meaning even the figures companies do publish understate the gaps.

The discrimination feedback loop: workforce to systems

The report's analytical contribution beyond cataloguing statistics is its argument that workforce composition and AI system discrimination are connected through a structural feedback loop. A homogeneous workforce develops systems that replicate its creators' assumptions and blind spots, with bias embedded not merely in datasets but in the organizational conditions under which data is collected, labeled, evaluated, and deployed. The report illustrated this feedback loop with three case studies prominent in the period immediately preceding publication.

Amazon's experimental resume-screening algorithm, tested internally and abandoned in 2018, had begun systematically downgrading applications from candidates who attended women's colleges and flagging resumes containing gender-associated terminology. Technical debiasing attempts had failed because the discrimination was embedded in historical hiring data generated by an industry that had systematically underhired women in technical roles — the model faithfully reproducing the patterns it had learned. Amazon Rekognition, the commercial facial-analysis service, showed gender and racial biases worse than comparable tools, failing disproportionately to recognise the faces of dark-skinned women while performing accurately on light-skinned men — findings the report situated alongside the Gender Shades audit published in 2018 by Joy Buolamwini and Timnit Gebru. Facebook's ad delivery algorithms produced discriminatory targeting outcomes without any advertiser intent: lumber industry jobs shown disproportionately to white men, cashier positions to women, taxi driver advertisements to Black users — discrimination embedded in the optimization functions the platform uses to predict ad relevance, which had learned those patterns from historical labor-market and consumer data that itself encoded prior discrimination.

The pipeline critique and twelve recommendations

The report's central polemical target is the "pipeline" framing dominant in industry diversity discourse — the argument that underrepresentation is primarily a supply-side problem caused by insufficient numbers of women and people of colour entering STEM, which will self-correct as educational pipelines diversify. The report argues that decades of pipeline-focused interventions have failed to move the numbers because they systematically ignore the demand-side conditions under which people exit: workplace cultures, power asymmetries, structural harassment, exclusionary hiring practices, unfair compensation, and tokenization dynamics that cause underrepresented workers to leave at higher rates than they enter. The pipeline framing is further critiqued as insufficiently intersectional: programs framed around "women in tech" have tended to privilege white women while leaving race, class, disability, and transgender identity outside their scope.

The twelve recommendations divide into two blocks. The first eight address workplace conditions: publishing compensation data by race and gender across all roles including contract workers; ending pay inequality and setting explicit equity goals; publishing transparency reports on harassment and discrimination complaints and outcomes; reforming hiring through targeted outreach to underrepresented communities; ensuring transparency across hiring, levelling, compensation, and promotion; increasing representation of women and people of colour at senior leadership specifically; tying executive compensation to diversity hiring and retention outcomes; and extending diversity requirements into academic AI spaces and conference programme committees. The final four address AI system bias through accountability mechanisms: tracking and publicly disclosing where AI systems are deployed and for what purposes; requiring rigorous lifecycle testing in sensitive deployment domains including pre-release trials, independent audits, and ongoing monitoring; expanding bias research beyond narrow technical debiasing to include wider social analysis of system context and power structure; and applying prospective risk assessment to evaluate whether specific systems should be built at all.

Position within the corpus

Discriminating Systems is the corpus's first dedicated AI Now Institute publication entry, closing a gap in the publications layer for the algorithmic-accountability movement's foundational institutional reports. It sits alongside Algorithmic Justice League's technical audit work — both documenting discriminatory outcomes in deployed AI systems and naming the same corporate actors — but occupies a structurally distinct position: where AJL's Gender Shades methodology demonstrates discriminatory system performance through technical measurement, Discriminating Systems operates at the level of political economy, arguing that discriminatory systems are downstream of discriminatory workplaces and cannot be remediated without intervention at that structural level. As a publication type it fills the workplace-AI-discrimination-by-gender-race slot in the publications layer, a structural gap the Synthesizer identified as distinct from the technical-audit slot the Gender Shades audits occupy. Within the AI Now Institute's own output, the report is the first of the Institute's dedicated single-topic publications on power in AI — a line it established ahead of the Disability, Bias, and AI report (October 2019) and that its annual AI Now Reports carried as a through-line. The report's reach into the US legislative process — Whittaker testified before the House Subcommittee on Research and Technology on 26 June 2019 — gives the corpus a publication-side connection for congressional reception of AI bias research, complementing the organisation-side connections through Algorithmic Justice League's contemporaneous congressional engagement on facial recognition legislation.