Black-Boxed Politics: Opacity is a Choice in AI Systems

Black-Boxed Politics: Opacity is a Choice in AI Systems is a long-read research report by Agata Foryciarz, Daniel Leufer, and Katarzyna Szymielewicz, with illustrations by Olek Modzelewski, published by the Panoptykon Foundation on 17 January 2020. The report advances the substantive argument that algorithmic opacity in artificial-intelligence systems used in essential public services and high-risk processes is a deliberate political and commercial choice rather than a technical inevitability — that "neither form of opacity, the technical or proprietary, is entirely inevitable" — and that the appropriate democratic response is to demand transparency about the design choices that produce opaque systems rather than to accept "the AI is a black box" as a stopping argument for civil-society oversight. It is the corpus's first Polish / Central-Eastern European publication anchor, and the named-byline counterpart on the Panoptykon publication line to the foundation's subsequent empirical-experiment register on platform recommender systems anchored on Algorithms of Trauma.

Argument and central framing

The report's central argument is that AI-system opacity is "a choice" — a deliberate settlement made by system designers, deployers, and political backers about how much transparency to permit — and that the public-debate framing of algorithmic systems as intrinsically uninspectable "black boxes" misrepresents how opacity is actually produced and conceals the political content of design decisions. The report's running formulation, that "neither form of opacity, the technical or proprietary, is entirely inevitable", reframes the question of AI accountability from a technical problem ("how do we explain neural networks") to a political problem ("whose interests are served by treating these systems as unexplainable, and on what terms"). The report's policy recommendation follows from the reframing: civil-society and democratic oversight should "demand transparency in the technical choices behind AI systems used to make decisions that affect humans so that we can see, and potentially challenge, the political decisions" — and "public authorities should not acquire AI systems from third parties in circumstances where the third party is unwilling to waive restrictions on information".

Three sources of opacity

The report develops the choice-not-inevitability argument through a taxonomy of three distinct sources of opacity that the public-debate framing of AI systems as singular "black boxes" runs together.

1. Technical opacity — the complexity of contemporary machine-learning systems (notably neural networks) that exceeds straightforward human inspection of internal weights, gradients, and decision pathways. The report's substantive move is to argue that technical complexity does not preclude meaningful transparency about design choices — the data selection, target-variable selection, evaluation metric selection, and fairness-trade-off acceptance that determines how the system behaves in deployment.
2. Proprietary opacity — the trade-secret and intellectual-property protections under which system developers and deployers withhold technical specifications, training-data documentation, and audit access from independent scrutiny, even where (as the report observes of EU-funded systems) the system has been built with public money or is deployed in high-stakes public processes. The report's position is that proprietary opacity is the most directly chosen of the three sources and the most directly addressable through procurement rules and disclosure requirements.
3. Definitional opacity — the report's distinctive third-source contribution: the very term "artificial intelligence" obscures what specific systems actually are, lending the rebranded statistical and rule-based systems behind contemporary "AI" applications a borrowed mystique of capability and inscrutability. Naming what a system actually is — a logistic regression, a decision tree, a transformer language model — is itself a transparency move that the AI-as-black-box framing forecloses.

The substantive contribution of the three-source taxonomy is that each source is addressable through a different transparency intervention: technical opacity through design-choice disclosure and audit access, proprietary opacity through procurement-rules and trade-secret-waiver requirements, and definitional opacity through precise naming and public technical literacy. The taxonomy supplies the conceptual structure on which Panoptykon's subsequent EU AI Act civil-society co-drafting work — the 30 November 2021 founding joint statement and the 12 July 2023 trilogue statement — rests, by reframing transparency-and-explanation requirements as legitimate political demands rather than technical impossibilities.

Case studies

The report tests the three-source taxonomy against three case studies drawn from AI-in-public-services contexts where the political stakes of opacity are highest.

• iBorderCtrl — the EU-funded Horizon 2020 "AI lie detector" border-control prototype that the report uses to anchor the proprietary-opacity case. The report's formulation is that "neither the European Commission nor the developers of iBorderCTRL have provided any evidence to suggest that there is any technical reason why we cannot inspect how their software works" — the canonical demonstration that public funding does not produce proportionate public-scrutiny access and that opacity here is a chosen settlement rather than a technical constraint.
• COMPAS — the U.S. criminal-justice recidivism-risk-scoring system that the report uses to anchor the technical-and-political-choice case on fairness trade-offs. The report's formulation is that "the system calibrated to achieve maximum accuracy will often not perform equally well in making sure that no group is discriminated against (as in the case of the COMPAS algorithm)" — the substantive demonstration that even with full technical inspectability, the fairness-versus-accuracy trade-off is a political choice that someone has to make on behalf of those subject to the system, and that the design choice cannot be reduced to a technical optimisation question.
• U.S. hospital healthcare-cost-prediction algorithm — the report's healthcare-resource-allocation case study, which addresses the proxy-variable selection problem in algorithmic systems. The report's formulation is that "the model systematically recommends healthier White patients over Black patients with higher medical need for enrollment" — the substantive demonstration that the choice to use healthcare-cost as a proxy for healthcare-need encodes the existing racial inequities of U.S. healthcare spending into the model's recommendations, and that the substitution looks technically neutral while being politically loaded.

The three case studies are arranged to illustrate that the three-source opacity taxonomy operates across multiple AI-in-public-services contexts and that the chosen-not-inevitable argument applies to AI systems in border control, criminal justice, and healthcare alike.

The Story of a Data Scientist

The report's distinctive expository device is a fictional narrative following Jasmine, a hospital data scientist working on patient selection for a health management program, through the stages at which she makes design choices — defining patient "need", selecting features, choosing evaluation metrics, accepting fairness trade-offs — that determine the political character of the deployed model. The narrative develops the report's argument inside the working register of an actual model-development process, showing that the political content of the system enters not through a single villainous design choice but through the accumulation of routine technical decisions each of which has a plausible technical justification. The substantive move is to make visible the human-design-choice scaffolding that the "AI is a black box" framing renders invisible — and to model the kind of decision-by-decision transparency the report's policy recommendations call for.

Authorship

The report is co-authored by three contributors whose joint register anchors the report's working synthesis of European digital-rights advocacy, AI-policy research, and algorithmic-fairness technical work.

• Katarzyna Szymielewicz — Panoptykon Foundation co-founder, President, and Advocacy & Strategy Director, and the corpus's Polish / Central-Eastern European Voice anchor on AI policy, surveillance advertising, and the Big Tech business model. Szymielewicz anchors the report's European digital-rights civil-society register and the connection to Panoptykon's institutional EU AI Act co-drafting track. The report is one of two flagship English-language essays on Szymielewicz's Medium author profile, alongside her earlier 14 November 2019 "A New Deal for Data" manifesto on platform-data-economy reform — see her Voice entry for the wider named-byline register through which her substantive framings have entered the European digital-rights field.
• Daniel Leufer — at the time of publication a 2019–2020 Mozilla Open Web Fellow hosted by Access Now, working on what would become aimyths.org, the AI-myths-and-misconceptions debunking resource that the Black-Boxed Politics "definitional opacity" argument fed directly into. Leufer holds a PhD in Philosophy from KU Leuven — the institutional context in which the KU Leuven Institute of Philosophy news-spotlight page on the report runs — and is now Senior Policy Analyst at Access Now in Brussels working on AI, facial recognition, biometrics, and augmented/virtual reality. Leufer anchors the report's philosophy-of-technology and AI-myths-debunking register and the connection to the Access Now European-policy AI-and-human-rights track.
• Agata Foryciarz — at the time of publication a Stanford Computer Science PhD researcher in the Health Policy Data Science Lab supervised by Sherri Rose. Foryciarz anchors the report's technical-algorithmic-fairness and health-policy-data-science register, including the U.S. hospital healthcare-cost-prediction case study and the technical scaffolding behind the "Story of a Data Scientist" narrative. Her academic-site biography names algorithmic fairness, AI transparency, and technology's potential harms as the cluster of research interests she carried into the report.

The report is illustrated by Olek Modzelewski. Foryciarz and Leufer are not in the corpus as Person or Voice entries; only Szymielewicz is reflected in the authors: frontmatter, in line with the Felicia Anthonio handling on Rising repression meets global resistance — Internet shutdowns in 2025, the Fabio Chiusi handling on Automating Society Report 2020, and the Neema Iyer / Chenai Chair / Garnett Achieng handling on Afrofeminist Data Futures.

Posture within the corpus

Within the corpus, Black-Boxed Politics is the publication-side anchor of the Panoptykon Foundation on the Polish / Central-Eastern European publications slate and the first Polish / CEE publication entry in the corpus. The org-side body identifies Panoptykon's three connected fronts — state-surveillance and Polish-court constitutional litigation; platform-accountability and algorithmic-systems research anchored on the Algorithms of Trauma report track; and European-policy co-drafting through EDRi — and the Black Box report sits at the conceptual hinge between the algorithmic-systems-research front and the European-policy front: it is the foundation's foundational long-read on the AI-and-human-rights substrate that the empirical-platform-recommender research and the EU AI Act co-drafting work both rest on.

In the corpus's publications slate, Black-Boxed Politics is the AI-opacity foundational-essay counterpart to the existing AI-and-human-rights foundational-essay register anchored by Stochastic Parrots (Bender / Gebru / McMillan-Major / "Shmargaret Shmitchell" on large-language-model harms), Unmasking AI (Buolamwini on facial-recognition algorithmic bias), and Gender Shades (Buolamwini / Gebru on commercial facial-recognition gender-skin-tone disparities). The four anchor the corpus's emerging foundational-essay register on the technical-political content of contemporary AI systems: Stochastic Parrots names the large-language-model harm cluster, Unmasking AI and Gender Shades name the facial-recognition algorithmic-bias cluster, and Black-Boxed Politics names the algorithmic-opacity-as-political-choice cluster against which the EU AI Act civil-society coalition has organised its transparency-and-explainability asks.

In the regional shape, Black-Boxed Politics installs the Polish / CEE leg of the corpus's foundational-AI-essay register alongside the existing Anglophone (US / UK) and Continental-Western-European (Germany / Brussels) legs. The corpus's existing AlgorithmWatch-anchored Continental-European register — Automating Society Report 2020 on the European-state automated-decision-making mapping — sits west of Panoptykon on the same Continental-European axis; Black-Boxed Politics is the Polish digital-rights NGO complement to the German watchdog-research-organisation comparative-state-mapping register, advancing the conceptual-political argument that the AlgorithmWatch mapping work documents on the empirical side.

The report's lasting movement-side contribution is that it supplies the conceptual ground for the civil-society position that the appropriate response to algorithmic opacity is regulatory transparency-and-explanation requirements rather than acceptance of the "AI is a black box" framing as a stopping argument — the working argument that has carried into the EU AI Act's transparency, risk-assessment, and public-register provisions and into the European civil-society field's broader posture toward AI accountability. In the corpus's terms it is the load-bearing Polish / Central-Eastern European publication anchor on the AI-and-human-rights publication line and the foundational long-read on which Panoptykon's subsequent platform-accountability publication-and-policy work is layered.