Introduction

The use of Artificial Intelligence (“Ai”)[1] systems to make and support decisions is increasing throughout various sectors of society. This is because machine learning (“ML”), a subset of Ai, facilitates a level of data processing, analysis, and decision-making precision that promises to revolutionise industries by maximising efficiency, cutting costs, and accelerating production.[2] For instance, Ai algorithms are increasingly used to make and support decisions with a legal or similarly significant impact for individuals in criminal justice,[3] employment,[4] finance,[5] surveillance,[6] law enforcement,[7] education,[8] in autonomous lethal weapons,[9] cyber-attacks and cyber warfare,[10] and the distribution of public services.[11] While on occasion this new level of accuracy and efficiency in decision-making can lead to significant improvements, it is accompanied by significant trade-offs, unprecedented risks and challenges of a novel and unique nature.[12] Notably, this “accuracy is not distributed equally among different demographics because of system bias.”[13] As such, one of the more pressing concerns in the context of governing Ai technology is the potential for Ai to perpetuate and amplify existing patterns of discrimination.

For instance, in February 2025, the Netherlands Institute for Human Rights, also known as the Dutch College for Human Rights, ruled that Meta’s Facebook algorithm unlawfully discriminated against women by disproportionately showing job advertisements aligned with historic gender stereotypes. The case, brought by NGOs Stichting Clara Wichmann and Global Witness, revealed that Meta’s algorithm displayed secretary roles to 85–97% female users and mechanic roles to 96% male users. In its ruling, the Dutch College applied Directive 2000/78/EC on Equal Treatment in Employment rather than newer digital regulations like the Digital Services Act (“DSA”) or Ai Act. This case is similar to the 2022 case of Real Women in Trucking v. Meta Platforms, Inc. which involves allegations of systemic gender and age discrimination in job advertisement distribution on Facebook, this time in California. The charge, filed with the USA’s Equal Employment Opportunity Commission by Real Women in Trucking, a nonprofit advocating for women in the trucking industry, similarly contended that Meta’s algorithms used to optimise advertising decisions, disproportionately steered job advertisements away from women and older individuals, in violation of federal anti-discrimination laws, including Title VII of the Civil Rights Act of 1964, and the Age Discrimination in Employment Act of 1967. The central allegation here is that even when advertisers requested their job ads be shown to all genders, Facebook’s ad delivery algorithm independently determined that trucking job advertisements were more relevant to men than women, resulting in discriminatory distribution. Advertisements for traditionally male-dominated jobs like truck driving or mechanics were shown overwhelmingly to men – up to 99%, while ads for traditionally female-dominated roles like administrative assistants were shown predominantly to women. At the time of writing, the case is in the investigatory phase with the EEOC.

In response to growing concerns around Ai governance, the EU adopted its landmark Artificial Intelligence Act (“Ai Act” or “the Act”) in 2024, becoming the first regional legal system to establish a legal framework for the governance of Ai.[14] Among other things, the Act identifies Ai system bias that is likely to lead to discrimination as a significant risk to fundamental rights in the context of Ai systems. Accordingly, it attends to bias through a multi-faceted approach comprising multiple and mutually reinforcing measures throughout the Act, aimed at minimising the risk of bias, either directly or indirectly.

Following a systematic review of the Ai Act, to identify provisions relevant to non-discrimination, this paper unpacks the Ai Act’s approach to algorithmic discrimination, concentrated at Article 10 of the Act, which the paper structures around four regulatory movements: the invoking of existing EU non-discrimination frameworks, the emphasis on bias and data quality, and establishing a data quality criteria, an exception to GDPR provisions on processing special categories of personal data, and finally, actor-specific obligations aimed at making the use and deployment of high risk Ai systems transparent and explainable. By critically examining these movements, this paper assesses the Act’s effectiveness in addressing the complex and evolving challenges of algorithmic discrimination. It finds that when it comes to addressing algorithmic discrimination, the Act adopts a strategy deeply rooted in technical requirements and processes, particularly for high-risk Ai systems. While this technical approach is necessary for tackling the unique ways Ai can perpetuate or create unfairness, it inherently limits the Act to a supporting function within the broader architecture of EU non-discrimination law.

The Proxy Problem

The proxy problem is a challenge in algorithmic fairness where seemingly neutral features are used as indirect stand-ins for prohibited grounds, leading to discriminatory outcomes in Ai systems. The nature of predictive Ai algorithms is to find connections between input data and target variables, regardless of those connections’ normative or legal character.[15] To the algorithms, it matters only that there is a predictive connection, for example between gender and future income, the potential explanations for the relationship between these connections (e.g., patriarchy, hetero-sexism, gender pay gap or history of exclusion) doesn’t matter at all.[16]

Sensitive grounds, or personal characteristics have a highly predictive or probative value in predicting humans’ future behaviour, because past and ongoing human discrimination has created unequal starting points for different groups, and different social structures have cemented this unequal status quo. Therefore, restricting access to highly predictive data points makes algorithms statistically less accurate in predicting future human behaviour or outcomes.[17] To compensate for this and maintain statistical accuracy, algorithms rely instead on a “proxy” or an indirect indicator of that prohibited ground or restricted sensitive data – thereby promoting the very outcomes the law seeks to avoid. In the discourse, this is called the “Proxy problem.”[18] For example, instead of making a prediction based on “gender” due to legal restrictions in the form of “prohibited grounds approach,” the algorithms will instead rely on a proxy or indirect indicator of gender like the applicant’s purchasing patterns, membership of a particular group, their name, communication style, or occupation from which to infer their gender. Thus, algorithms will always bypass a legal restriction where the data restricted is predictive of future outcomes.

Ai has no capacity to know ethics, principles or norms in the same way conscious human minds do. As notes by Deck et al., “legal concepts relying on flexible ex-post standards and human intuition are in tension with the mathematical need for precision and ex-ante standardisation.”[19] Unlike human decision makers, Ai does not depart from any theory or hypothesis about what types of characteristics may prove useful for predicting the target variable. Rather, it uses “brute force” to learn from scratch which attributes or behaviours predict the outcome of interest.[20] As a data-driven system, Ai can understand hard, individual technical parameters, not principles and norms, such as equality and non-discrimination.[21] While the “prohibited grounds approach” can target intentional proxy discrimination by human actors, it fails in the context of proxy discrimination by Ai data-analysing algorithms. I submit that the approach finds more success with human decision makers, because unlike Ai, human decision makers are conscious entities, with cognitive qualities that allow a capacity to “know” (to have direct experience and awareness of) the normative and ethical reasons between different connections, and by extension, the legal and ethical nature of those connections, too.[22] I submit that the law implicitly replies on this quality on the part of the human subject or decision maker – a capacity to “understand” the underlying normative reasons, parameters and spirit of the prohibition. This way, a human decision maker will understand that a prohibition on gender will necessarily include all other indirect indicators of gender, along with the meta-ethics thereof.

Moreover, as Ai algorithms continue to evolve, they become more sophisticated and complex, and become increasingly adept at identifying and relying on an even broader array of proxies or indirect indicators of prohibited grounds. These proxies grow increasingly more subtle, less obvious and less intuitive to the human mind, often appearing disconnected.[23] For instance, the presence or absence of certain mobile apps, the frequency of software updates on devices, series or music preference, keyboard typing speed, frequency of contact with customer service can all be used to indirectly indicate individual characteristics that are otherwise restricted, such as a person’s age, income levels (class), cultural background or risk aversion.

Proxy discrimination is “inherent and ubiquitous” in both algorithmic and human decision-making. In inductive reasoning, we (humans) often rely on some characteristics to “stand in” for other deeper characteristics to which we often don’t have access.[24] The same is true for algorithmic inductive reasoning. Take for example of object recognition algorithms, they are never in contact with the actual objects they aim to identify. Here, the algorithm relies rather on images, where some collection of pixel values will be a proxy for some other feature, such as texture or shape, which in turn act as a proxy for the target attribute, such as a cat, a missile, or a pedestrian.[25] In fact, it is common to train ML algorithms on proxy characteristics that are easier to measure than the characteristics or attributes we want the system to predict. As such, discrimination that results from algorithmic use of proxy characteristics cannot be avoided or mitigated using overt filtering techniques such as those of the “prohibited grounds approach” employed by existing EU and national non-discrimination framework. As Goodman and Flaxman put it, “it is widely acknowledged that simply removing certain variables from a model does not ensure predictions that are, in effect, uncorrelated to those variables.”[26] This is because the algorithm will instead rely on a proxy of the prohibited ground, thereby resulting in the same discriminatory outcome. Regrettably, this fundamental challenge to non-discrimination remains under-examined under the EU Ai Act.

Unpacking Ai Act’s treatment of Algorithmic Discrimination

In its fundamental nature, the Ai Act is an enhanced product safety legislation, following a mixed regulatory approach combining product safety and fundamental rights. It adopts a product safety legislative approach to managing risk – the risk-based approach,[27] wherein risk is defined as “the combination of the probability of an occurrence of harm and the severity of that harm.”[28] This approach shapes all obligations and requirements in the Act, bending them towards a fundamentally risk-based orientation, focusing on risk produced by Ai systems. Typically, product safety legislation targets risks to health and safety; however, given Ai’s impact on society, the Ai Act includes an additional risk dimension; it attends, in addition, to risks to fundamental rights – becoming an enhanced product legislation.[29]

As made clear repeatedly in the Act, one of its many objectives is the prevention of discrimination produced by Ai systems or algorithmic discrimination. The preamble of the Act consistently emphasises the principle of non-discrimination and acknowledges the associated risks in the context of Ai. However, discrimination remains a peripheral objective in the Act, receiving only limited, indirect attention from the Act. With this, the Act positions itself in a supporting, rather than primary role when it comes to the governance of non-discrimination, refraining from duplicating existing Union non-discrimination frameworks. The core legal definitions of prohibited discrimination, the grounds on which it is prohibited, the burden of proof in legal proceedings, and the remedies available to victims of discrimination are all contained within existing Union non-discrimination directives and related jurisprudence. These frameworks provide the comprehensive legal infrastructure for challenging and redressing discrimination, including that facilitated by Ai. The Ai Act does not replicate this infrastructure, nor does it establish a parallel system for adjudicating discrimination claims. These established frameworks remain the primary legal avenues for individuals to challenge discriminatory treatment, including that which is caused by ML decision systems. Instead, the Act explicitly invokes “discrimination prohibited under Union law” (Article 10(2)(f)), signalling that the legal standard for what constitutes discrimination remains firmly within the existing body of EU non-discrimination directives.

The Act’s contribution is to provide specific rules tailored to the unique technical characteristics of Ai systems that are likely to give rise to discriminatory risks. To that end, the Act has identified a number of specific risks, including risks of “biases that are likely to affect the health and safety of persons, have a negative impact on fundamental rights or lead to discrimination prohibited under Union law,” as stated in Article 10(2)(f).[30] In the latter section, the Act creates a link between the different concepts of “bias” in computer science and “discrimination” in law. It regulates at the level of bias, to support efforts to prevent discrimination. Kristof Meding’s analysis offers us a useful framework for understanding Act’s attempt at regulating bias to support the achievement of non-discrimination in Ai systems.[31] Meding argues that there is difference between the concepts of “bias,” and “algorithmic fairness” as understood in computer science, and “non-discrimination” within the domain of law. Terms like “bias” and “fairness” in the computer science context do not fully align with their legal counterparts. Bias in a dataset, for instance, might be a statistical imbalance or disparity, but it translates into legal discrimination only if it leads to unjustified differential treatment on the basis of a prohibited ground. While both domains are broadly moving towards the same goal of attending to unfair outcomes, their approaches diverge. Algorithmic fairness and bias mitigation and debiasing techniques are rooted in computer science and primarily focuses on the formal, quantitative technical aspects of detecting, preventing, and mitigating statistical imbalances within data sets used to train algorithms, relying on statistical and mathematical metrics, formulas and rules.[32]

Non-discrimination, in contrast, is premised in normative, value-driven principles. It is concerned with protecting individuals and groups from objectively unjustifiable differential treatment based on selected protected grounds such as race, ethnicity, gender, religion, disability, age, or sexual orientation. The test for discrimination under non-discrimination frameworks involves analysing the outcome of a practice or decision. The jurisprudence of the ECtHR, sets the test as follows; “a difference in the treatment of persons in relevantly similar situations… is discriminatory if it has no objective and reasonable justification; in other words, if it does not pursue a legitimate aim or if there is not a reasonable relationship of proportionality between the means employed and the aim sought to be realised.”[33] In Belgian Linguistics, the ECtHR established that an objective and reasonable justification is established where  the measure in question has a legitimate aim and there is “a reasonable relationship of proportionality between the means employed and the aim sought to be realised”[34]

The Act’s primary strategy, as reflected in Article 10 and other related provisions concerning data quality and risk management, leans heavily on a technical paradigm of bias detection and bias mitigation, which in practice occurs through formal, mathematical fairness metrics, debiasing techniques, etc which employ quantitative methods and metrics to assess and correct imbalances in datasets and algorithmic outputs. The focus is on ensuring that training, validation, and testing data are of high quality, relevant, and sufficiently representative, and that measures are in place to identify and address statistical biases within these datasets that are likely to impact non-discrimination prohibited under Union law

This approach is categorically not sufficient on its own to guarantee non-discrimination. Bias arises not just from biased data, but also from the design of the algorithm itself, the choice of objectives, the context of deployment, and the interaction of the Ai system with existing societal inequalities. Accordingly, the Act identifies 3 sources of this risk of discrimination: bias in the data used to train the Ai system;[35] the design of the algorithm itself;[36] and the way the Ai system is deployed and used, including the context.[37]  However, and to the point of this paper, bias arises from a more fundamental location, ML’s confinement to formal operational logic and techniques, which cannot simulate components of non-discrimination and fairness. As Meding and other experts in critical algorithmic discourse highlight,[38] non-discrimination and equality are not merely a matter of statistical parity or the absence of bias in data. It is a normative concept rooted in dignity, and equality principles.[39] Non-discrimination involves normative judgments about fairness, proportionality, and the legitimacy of differential treatment in specific contexts, which cannot be reduced solely to statistical properties of the input data. This more fundamental limitation explains and accounts for algorithms’ reliance on proxies, and remains unresolved even with hygienic, representative data, fairness metrics or debiasing techniques.

Technical fairness metrics, while useful for identifying potential biases in data sets, cannot capture the full spectrum of discrimination, or its principle of substantive quality, as understood in law juridically or jurisprudentially. Different, often contradictory, fairness metrics exist, however, research is already finding that they are insufficient to meet the requirements of non-discrimination law – for the perhaps simple reasons that fairness or equality cannot be simulated through formal techniques like fairness metrics in ML.[40]

As such, with this technical approach, the Act can only ever play a supporting role to non-discrimination frameworks. Bias in an Ai system’s data or design is a technical phenomenon that can cause discrimination. However, discrimination itself is a legal concept defined by its impact and lack of legal justification. The Act’s technical measures target the cause (bias), but the legal frameworks for non-discrimination govern the effect or discriminatory outcomes. A purely technical approach, even one that can process sensitive data for bias detection, cannot fully capture or govern these complex socio-technical interactions that lead to discriminatory outcomes in the real world.

In essence, the Ai Act acts as a layer of Ai-specific regulation that imposes upstream obligations on providers of high-risk Ai systems to reduce the likelihood of discrimination occurring. However, the ultimate legal assessment of whether discrimination has taken place, the grounds on which it is prohibited, and the available remedies are still governed by the foundational Union non-discrimination frameworks. To show it, the Act itself does not require Ai system to be unbiased or discrimination to be eliminated. By refraining from creating a new, comprehensive legal framework for non-discrimination in the Ai context, the Act avoids duplicating the extensive body of existing Union non-discrimination law. Rather, it provides tools and obligations[41] that are necessary to identify and address these Ai-specific technical risks contributing to discrimination.[42]

While algorithmic fairness and bias mitigation provides valuable, even necessary tools they cannot be a direct substitute for or equivalent to the normative requirements of legal non-discrimination frameworks. In the context of algorithmic discrimination, the Act cannot make any claim of solving the problem, its provisions and approach is limited to a supporting role, supporting existing non-discrimination law frameworks. Below I unpack the Ai Act’s treatment of algorithmic discrimination into four key movements, and offer a critical analysis of each, in turn, demonstrating their supporting, rather than primary function in the governance of algorithmic discrimination.

A. Movement 1 Invoking existing EU non-discrimination law

As part of its strategy of achieving consistency with, and avoiding duplication within the EU regulatory landscape,[43] the Act invokes pre-existing EU frameworks that prohibit non-discrimination. At Recital 45[44] to the Preamble, it explicitly provides that it does not affect the application of existing Union law prohibiting discrimination, thereby confirming that existing non-discrimination protections remain in place in the context of Ai.[45] For instance, at Article 10(2)(f), the Act creates an obligation for providers of high risk Ai systems to examine high risk Ai systems for “possible biases that are likely to … lead to discrimination prohibited under Union law.”[46]

Those prohibitions, under existing law, can be found for instance in various national laws on non-discrimination, EU directives and regulations, as well as in European regional human rights instruments. They include Article 14 of the European Convention of Human Rights,[47] Article 21 of the Charter of Fundamental Rights of the European Union,[48] Article 2 of Directive 2000/43/EC on racial or ethnic origin;[49] Article 1 of Directive 2000/78/EC equal treatment in employment and occupation;[50] Article 4 to Directive 2006/54/EC;[51] Article 4 to Directive 2004/113/EC;[52] and Article 1 and 2 of Directive Proposal (COM(2008)462).[53] Each of these prohibitions set out a framework for the prohibition of discrimination on the basis of selected prohibited grounds, to establish a uniform minimum level of protection within the State, EU and European continent respectively. These lists of prohibited grounds present a “societal rejection” of those grounds as acceptable foundation for differentiation in decision-making.[54]

The Act fully accepts and relies upon the definitions, scope, and principles of discrimination as they currently exist within the Union legal order. The technical requirements imposed by the Act, particularly on high-risk Ai systems, are designed to provide duty-holders (like Ai developers and deployers) with specific technical obligations and processes (such as robust data governance and bias mitigation) that help them comply with their pre-existing legal obligations under non-discrimination law. The Act offers Ai-specific methods and standards to reduce the risk of engaging in conduct that would already be considered discriminatory under existing laws. The Act’s preventative focus on identifying and mitigating bias at the technical level is a strategy aimed at avoiding outcomes that would trigger a violation of existing non-discrimination law. It’s about building Ai systems in a way that respects and upholds the principles already enshrined in Union law, rather than defining those principles anew. This ensures coherence within the EU legal system and reinforces that existing non-discrimination laws remain the primary and comprehensive framework for defining and addressing discriminatory conduct, regardless of whether it is facilitated by Ai.

B.  Movement 2: A focus on bias and data quality

Article 10(1) of the Act establishes a heightened data quality criteria for those Ai systems classified as high risk, to the extent that they rely on data and ML techniques.[55] This quality criteria consists of 3 requirements:

1. The “[t]raining, validation and testing data sets shall be subject to data governance and management practices” that must relate to 8 aspects outlined at subparagraph 2(a) to (h).[56] Key of these aspects are items (f) and (g), which respectively provide that these data governance and management practices must concern “examination in view of possible biases that are likely to … lead to discrimination prohibited under Union law, especially where data outputs influence inputs for future operations” (feedback loops); and that they must include “appropriate measures to detect, prevent and mitigate possible biases identified according to point (f).” This is a procedural obligation, not a substantive obligation, meaning, to comply fully with this obligation, it is sufficient for a provider or deployer to demonstrate that an appropriate data governance and management practice is in place, and that it relates to the 8 aspects outlined at subparagraph 2(a) to (h).[57]
2. The training, validation and testing data sets must be “relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose.” And further that the data sets must be “statistically representative” with regards to persons or groups of persons in relation to whom the high-risk Ai system is intended to be used.[58]
3. Data sets must be contextual, taking into account “the characteristics or elements that are particular to the specific geographical, contextual, behavioural or functional setting within which the high-risk AI system is intended to be used.”[59]

The Act’s focus on bias and data quality is premised on the understanding that Ai systems (re)produce discriminatory outcomes when trained with biased and inaccurate data. Garbage in, garbage out.[60] At Recital 67 the Preamble recognises that “biases can, for example, be inherent in underlying data sets, especially when historical data is being used, or generated when the systems are implemented in real-world settings. The obligation that high-risk Ai systems be developed using training, validation, and testing data sets that “meet the quality criteria”,[61] the Act establishes a technical pillar supporting the legal principle of non-discrimination. The precise nature of the “bias” addressed by Article 10(2) remains a point of contention. As Kristof Meding critically observes, the focus may be overly narrow; “It seems that the regulators had a more technical definition of bias in mind, focusing on the diversity of training data in different dimensions compared to social, ethical, or structural biases.”[62] Furthermore, he highlights a crucial consequence; “This makes it very hard to determine the regulatory content” of Article 10(2), pointing to the resulting ambiguity for developers and deployers seeking to comply.”[63]

Meding also examines the scope of the phrase “that are likely,” arguing that it modifies all subsequent conditions in the sub-paragraph. As Meding notes, it contrasts with the definition of “risk” found elsewhere in the Act, which involves a balancing of both likelihood and severity of harm. By suggesting that the “likely” threshold in Article 10(2)(f) primarily concerns the probability of the adverse effects manifesting, Meding implies that this specific obligation for bias examination is triggered by a likelihood assessment, different from a comprehensive risk evaluation triggered by the additional element of  severity.[64] “Compared to the risk in Article 9(2) AIA, in Article 10(2)(f) AIA, the severity of the harm is not taken into account, only the likelihood.”[65]

The requirement to address biases leading to outcomes “that are likely” to negatively affect fundamental rights or constitute discrimination presents an “open question” regarding the necessary threshold – certainty or probability.[66] Meding lean towards a probabilistic reading, arguing it “seems more consistent” with the Act’s broader focus on risk, particularly when compared to Article 9(2)(a)’s explicit mention of risk to fundamental rights.[67] They find additional support for this interpretation through linguistic comparison, noting that in translations like the German text, “it is more evident that the wording ‘that are likely’ applies to all conditions.”[68] This suggests the intent was likely to capture probable harms, aligning with a risk-management framework. Adopting this view means Article 10(2)(f) mandates addressing not just definite, but also probable, discriminatory outcomes or rights infringements stemming from data bias. “the severity of the harm is not taken into account, only the likelihood.”[69]

To the extent that it contrasts with the use of risk in other articles, for instance Article 9(2)(a), “unnecessary regulatory uncertainty,” noting that it is “unclear why the legislator opted for the difference in applicability between Article 9 AIA and Article 10 AIA.”[70]

What Article 10 offers in a supporting function to non-discrimination law is the provision of enforceable technical standards aimed at preventing discrimination upstream. Article 10, offers concrete, Ai-specific requirements that, if met, reduce the likelihood of an Ai system producing results that would violate those existing legal standards. However, even its supportive function is ultimately limited; non-discrimination law requires the elimination and reduction of discrimination. The measures employed in the Act cannot guarantee non-discrimination. An Ai system can comply with Article 10 and still result in discrimination. The fundamental reason why an Ai system can comply with the technical requirements of Article 10 and still result in discrimination lies in the inherent difference between technical bias detection at the development stage and the complex, context-dependent nature of legal discrimination in real-world deployment. Therefore, while Article 10 and similar technical provisions in the Act are necessary for building Ai systems that are more likely to be fair and non-discriminatory by addressing technical biases at the source, they are not a fail-safe. The possibility remains that even an Ai system developed in full technical compliance with Article 10 could, due to the inherent complexities of Ai and its interaction with society, lead to discriminatory outcomes.

First, this is because all data is “dirty data.” As Meredith Broussard poignantly puts it, “[A]ll data is dirty. All of it.”[71] There is no such thing as “clean data” or “unbiased data.”[72] The phrase itself is an oxymoron. This is because all data is a product of human language, experiences, cultural artifacts, human interpretation, selection and decision-making at some stage throughout the data life cycle, whether at collection, categorisation, or presentation.[73] As Eaglin contends, data is not just found but selected and crafted with particular value judgments. To illustrate this, she points to the example of input data in ADM systems used to assess risk of recidivism, noting that the data centres on “policy questions about who should be considered a risk and how much risk society tolerates.”[74]

Consider the process of data cleaning or processing for example. Once data is collected, it goes through a ‘chewing’ process, wherein all data is standardised to the algorithm’s training requirement, to facilitate the interoperability of data within quantitative frameworks and tools. Data that deviates from the norm or exhibits unique characteristics is often filtered out or refined by human data scientists or programmers to align better with the rest. The process also removes significant portions of the original formatting and context, introducing a profound decontextualization of the data, which in turn bears in the interpretation and understanding of this data.[75] As Solovo contends, “[w]hen qualitative data is removed, leaving just quantifiable data, the nuance, texture, and uniqueness of individuals is lost…decisions are being made about people based on a distorted picture of them.”[76]

Where the algorithms require labelled data, the individuals labelling the data bring their own biases and interpretations to bear, which will subsequently be learned by the algorithm. Further, the Act itself recognises at Article 10(2) that certain design choices or features chosen to represent the data can further introduce bias into the algorithm, for instance where important features are omitted or where irrelevant features are given undue weight.[77] To this point, Cathy O’Neil correctly notes that development of algorithmic models involves selective attention to certain data while excluding others, – highlighting the inherent subjectivity involved in the development of algorithmic systems more broadly. To O’Neil, “[t]hose choices are not just about logistics, profits, and efficiency. They are fundamentally moral.”[78]

Those fulfilling the human oversight role or interpreting the algorithmic output/results also bring their own pre-existing conscious and unconscious biases; or if they lack a deep understanding of the algorithm’s limitations, they might draw biased conclusions.[79] Barocas and Selbst succinctly summarise the issue: “Big data claims to be neutral. It isn’t.”[80] As the authors point out, machine learning relies on data collected from society. Consequently, to the extent that society embodies inequality, exclusion, or other forms of discrimination, these biases will inevitably be reflected in the data.

In this way then, every data set carries with it the fingerprints of societal, cultural, and personal biases of those who played a role in its life cycle.[81] In what she calls the “input fallacy,” Talia Gillis reminds us that efforts to remove or restrict access to legally prohibited grounds, such as race, gender or sexual orientation from input data, does not eliminate discrimination and bias in Ai outputs, observing correctly that these prohibited grounds can still be inferred from other available data about the individual, demonstrating that biases are intricately woven into the fabric of the data, beyond direct identifiers.[82] The takeaway here is that the data driving or Ai decision systems, while seemingly neutral, are normative, often reflecting biases, stemming from historical inequalities, societal norms, and the subjective choices made about which data to proceed without, which weightings to place, or variables to set. This reality undermines the idea that data can be neutral or unbiased.

The Act’s primary focus on data quality undoubtedly expands the legal framework for safeguarding non-discrimination in meaningful ways, however, it is clear that is not sufficient against the challenge of proxy discrimination. By focusing on technical checks for bias as a solution to discrimination, the Act risks reinforcing the misleading notion that it is possible to eliminate bias from Ai systems. It is not.[83] Data sets that are of “high quality”, “sufficiently representative,” “free of errors and complete”[84] are not sufficient to erase biases in data sets. The most that can be hoped for is a minimisation of system bias, not its elimination. Non-discrimination law sets a higher bar, requiring not merely the reduction of risk, but the elimination or effective mitigation of discriminatory effects and treatment experienced by individuals and groups based on protected characteristics. Moreover, focusing heavily on input bias is not sufficient to detect and prevent all forms of discrimination. Discriminatory outcomes can arise not only from biased data but also from proxies. Thus, working at the input level is necessary of course, but it is not sufficient.[85]

C. Movement 3: A new exception to GDPR

Third, Article 10(5) of the Act opens a marked exception to Article 9(1) the GDPR, which prohibits the processing of special categories of personal data. The GDPR is a technology-neutral legal framework; we see this in its expansive definition of “processing” which encompasses nearly all operations performed on personal data. As such, it is clear that the GDPR’s provisions extend to Ai systems, to the extent that personal data is at any point processed during the Ai system’s lifecycle.[86] To that end Article 9(1) of the GDPR prohibits, as a general rule, the processing of special categories of personal data, which is defines are those data points that “[reveal] racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership, and the processing of genetic data, biometric data for the purpose of uniquely identifying a natural person, data concerning health or data concerning a natural person’s sex life or sexual orientation.”[87]

Article 10(5) of the Ai Act creates an exception to this prohibition, to allow providers of high-risk systems, in exceptional cases, to process special categories of personal data where it is strictly necessary for the purpose of ensuring bias detection and correction. Guadino calls this Article a “paradigm shift” in the governance of special categories of personal data, remarking optimistically that “[the] paradigm shift is an expression of a fundamental optimism that our social reality can be improved in a sustainable manner through properly regulated AI.”[88] Article 10(5) provides a technical tool – a legal mechanism to process sensitive data – specifically for the purpose of fulfilling a requirement of bias detection and correction under Article 10(2)(f) and (g), that is explicitly linked to preventing discrimination prohibited under Union law.

The rationale of Article 10(5) is clear; you cannot effectively detect or correct biases against specific groups if you cannot analyse data related to those groups. Primarily, the Ai Act introduces this exception because system biases are not, in certain circumstances, detectable using only non-sensitive data.[89] Examining special category data can help reveal these biases, by allowing the limited and controlled processing of special category data.[90] In theory the Ai Act aims to enable developers to uncover and address these hidden biases, making them easier to mitigate. Processing special categories of personal data can provide a more nuanced understanding of how Ai systems impact different groups. Deck et al., writing on the implications of the Ai Act for non-discrimination law and algorithmic fairness contend that the development of fair Ai systems necessitates access to sensitive demographic data to identify and mitigate biases that may correlate with, and act as proxies for, prohibited grounds such as race and gender, thereby moving beyond the limitations of “fairness through unawareness.”[91] The authors contend that Article 10(5) of the Ai Act directly addresses this conflict by providing a legal basis for the exceptional processing of special categories of personal data where strictly necessary for the purpose of detecting and correcting bias in high-risk Ai systems, provided adequate safeguards are in place.[92]

There is evidence suggesting that allowing algorithms access to personal sensitive data can help detect and mitigate system biases and discriminatory outcomes.[93] In fact, existing research supports the view that permitting algorithms to access sensitive personal data, under strict and exceptional conditions, can be instrumental in identifying and mitigating systemic biases which lead to discriminatory outcomes. Several corporations have also developed tools and frameworks that leverage sensitive personal data, with appropriate safeguards, to detect and mitigate biases in Ai systems. IBM’s Ai Fairness 360 toolkit,[94] Google’s What-If Tool,[95] and Microsoft’s Fairlearn are prime examples of such initiatives.[96] These tools provide developers with the means to assess and improve the fairness of their Ai models by analysing their behaviour across different groups and identifying potential sources of bias.

Limitations

Article 10(5) provides a necessary enabler for tackling the proxy problem at the data level, but it is likely not sufficient to fully combat it on its own. Once potential proxies are identified, developers can then assess if these features are contributing to biased outcomes within the dataset or during model training, in line with the requirements of Article 10(2)(f). However, Proxies are subtle, context-sensitive and therefore dynamic, involving non-linear correlations between multiple seemingly innocuous features and a protected characteristic.

A feature that acts as a strong proxy for a protected characteristic in one application context or geographical area might not do so, or might proxy a different characteristic, elsewhere. For instance, certain linguistic patterns might correlate with origin in one region but not another. An Ai system trained and validated for bias using data from one specific context, relying on the ability to process sensitive data under Article 10(5) to identify proxies within that dataset, might still exhibit discriminatory behaviour when deployed in a different context where the proxy relationships are altered. The input analysis performed under Article 10 provides a snapshot valid for the specific dataset and context of development, not a universal guarantee against proxy-driven discrimination in all potential deployment scenarios. Identifying all such complex proxies in high-dimensional datasets can be technically challenging, even with access to sensitive data. The proxy problem is not just about the data itself, but also about how the algorithm uses the features.

Safeguards

Further, this processing of special categories of personal data must be in accordance with the obligation for data governance and management practices outlined at sub paragraph 2 of the Article, specifically points (f) and (g).[97] The Act makes this exception “subject to appropriate safeguards for the fundamental rights and freedoms of natural persons,[98] – including requirements of necessity; technical limitations on the re-use of personal data (purpose limitation), privacy-preserving measures and cyber security; strict controls and documentation of the access of this data, confidentiality; 3^rd party access, transmission and transfer prohibitions; and requirements for deleting the data once the bias has been corrected or once data retention period has expired (data minimisation).[99] This is in addition to the provisions set out in Regulation (EU) 2016/679, Directive (EU) 2016/680 and Regulation (EU) 2018/1725. Finally, the Act requires the keeping of records of processing activities, as well as the reasons why the processing of special categories of personal data was strictly necessary to detect and correct biases, and why that objective could not be achieved by processing other data.”[100] Here the Act assumes that what providers disclose will be true and correct. However, even with this, the approach raises serious concerns about risks to the privacy and data protection regime, because of Ai’s exponential capacity for pattern recognition and association, which offers an expanding capacity for data to be combined in a way capable of reliably (re)identifying, classifying and clustering individuals based on seemingly unconnected, non-personal pieces of data.[101]

Even with the ability to process sensitive data for bias detection in inputs, the gap between identifying input bias and preventing discriminatory outputs persists. An Ai system might process data containing special categories of personal data, and developers might use this to mitigate statistical biases in the training data. However, the system’s behaviour upon deployment, influenced by algorithmic design and context, could still lead to discriminatory outcomes that were not fully eliminated or foreseen during the input-focused bias correction phase, or that were introduced by the phase itself. Moreover, there are compelling incentives for providers to exploit this exception. The ability to process special category data can provide a competitive edge. By analysing this data, providers can develop more accurate and nuanced Ai models, potentially leading to better performance and increased market share.[102] Processing existing special category data under the exception can be a cost-effective way to improve Ai models without investing in additional data collection and labelling efforts, which can be expensive.[103] In more exceptional circumstances, it may incentivise providers to classify their Ai systems as high-risk even if the risks associated with their systems are not significant, or intentionally engineer conditions of bias that make it seem necessary to gain access to this exception and process special category data.

Consider for example enforcement actions against the utilisation of Ai systems by EU data protection authorities, before the adoption of the Act. These enforcement actions are based on a range of issues, including a lack of transparency, lack of legal basis to process personal data or special categories of personal data, failure to fulfil data subject rights, automated decision-making abuses, and data accuracy issues. The most notable are Italian DPA’s temporary ban on OpenAI’s ChatGPT,[104] and its fine against Deliveroo’s Ai-enabled automated rating of rider performance,[105] and the French DPA’s fine against Clearview Ai for scraping billions of photographs from the internet, including social media platforms, to create a vast database for facial recognition purposes.[106] These already demonstrate a propensity by Ai providers and deployers of exceed the lawful bounds in processing personal data and special categories of personal data, where it’s in their business interest to do so.

Moreover, research illustrates that these safeguards are already proving insufficient to protect privacy of personal data in the context of algorithmic data processing and decision making.[107]  As regards the first safeguard of necessity, the Act assumes here that the promoted alternative data types (synthetic or anonymised data) can adequately replicate the complexities of real-world data, which is often not the case.[108] Moreover, even though necessity is subjective and open to interpretation, the Act does not provide any guidance or criteria for making this determination, leaving room for potential misuse or abuse of this provision by providers. A more stringent and clearly defined necessity standard, coupled with robust oversight mechanisms, is required given the risks of Ai processing of personal data. Regarding technical limitations, the effectiveness of this safeguard hinges on the robustness of the technical measures implemented and the ability to enforce these limitations. The dynamic nature of Ai, with models constantly evolving and being integrated into new systems, poses a challenge in ensuring that the data remains confined to its intended use.[109] Additionally, the Ai Act does not explicitly define what constitutes “technical limitations,” leaving room for interpretation and potential loopholes that could be exploited.

Act mandates the use of “state-of-the-art security and privacy-preserving measures,” including pseudonymisation, to protect special categories of personal data. Studies are already demonstrating that these measures are not effective in the face of Ai capabilities.[110] Pseudonymisation refers to the process of substituting original identifiers with fictitious identifiers, commonly known as pseudonyms. While these measures aim to enhance privacy, their effectiveness can be questioned in the context of rapidly evolving Ai technologies.[111] As Boudolf notes for instance, “researchers recently succeeded in reconstructing both pixelized and blurred faces by making use of neural networks.”[112] Ai-powered neural networks can be employed to reverse-engineer anonymised data and potentially re-identify individuals. The term “state-of-the-art” is inherently fluid, as what is considered cutting-edge today may quickly become outdated. This brings into question the long-term viability of these measures in safeguarding sensitive data against emerging threats that arise with more sophisticated techniques. Pseudonymisation, while a valuable privacy-enhancing technique, does not completely anonymise data. As Ai-powered de-anonymisation techniques become more sophisticated, the risk of re-identifying individuals from pseudonymised data increases.

Moreover, the nature of Ai models, especially ML models that continuously learn and adapt, raise questions about the feasibility of complete data deletion. As Shah and Rishav note, “records in a database become interdependent” and the deleted data’s influence remains subliminally in the remaining data due to this interdependence.[113] Notably, the Act does not specify the duration of the retention period, leaving it open to interpretation by Ai providers, and can lead to situations where data is retained for longer than necessary, increasing the risk of unauthorised access or misuse. In the circumstances, a more stringent approach is warranted. Furthermore, these safeguards primarily focus on technical measures, potentially overlooking the human element in data breaches. The effectiveness of “strict access controls and documentation” hinges on the consistent and rigorous implementation of these protocols by all authorised personnel. However, human error, negligence, or even malicious intent can undermine these safeguards. The Act does not address the potential for insider threats or the need for ongoing training and awareness programs on the handling of sensitive personal data in ML models and about evolving security risks.

D. Movement 4: Transparency and Explainability

While technical bias mitigation measures, such as those in Article 10, focus on prevention at the development stage, transparency and explainability provide essential mechanisms for scrutiny and challenge after an Ai system is deployed. Thus, transparency is relevant for algorithmic discrimination when it comes to the enforcement of obligations to eliminate bias and discrimination in Ai systems. Transparency requirements under the Ai Act, particularly for high-risk Ai systems, serve as a foundational support for non-discrimination by addressing the “black box” problem associated with complex algorithms. It is a necessary first step for individuals and oversight bodies to even identify that a potentially discriminatory outcome might be linked to an Ai system. Without knowing that an Ai system was involved, or having basic information about its function, it would be exceedingly difficult to even begin investigating a suspected case of algorithmic discrimination. Transparency lifts a veil, enabling the possibility of legal contestability of algorithmic outcomes, including those that are discriminatory.

The right to an explanation, then, support individuals’ ability to receive meaningful explanations for decisions made by high-risk Ai systems that affect them, often indirectly facilitating the exercise of remedies under existing non-discrimination law. The ability to understand the key factors or reasons behind an adverse decision is crucial for an individual to determine if unlawful discrimination may have occurred and to effectively challenge that decision through appropriate legal channels. Explainability reduces the opacity of algorithmic decision-making, making it possible to scrutinise the basis for differential treatment. In the absence of such transparency, it becomes impossible to evaluate or adjudicate the outcomes of an Ai system, thereby hindering the identification of discriminatory patterns against any specific group or individual. The Act acknowledges this necessity, recognising the importance of transparency for both persons affected by Ai decisions and for judicial and oversight bodies responsible for adjudicating algorithmic discrimination claims. Nevertheless, in attempting to balance transparency with the protection of intellectual property and confidentiality, the Act fails to mandate a sufficient level of transparency in Ai systems to achieve its objective of upholding non-discrimination.[114]

1. Right to an explanation

Enforcement has consistently been a significant challenge in the context of non-discrimination law, especially in jurisdictions that primarily rely on individual litigation for the enforcement of the right.[115] Even under ordinary circumstances, persons affected by a discriminatory decision or outcome must contest with considerable difficulties in identifying, proving, and preparing a competent complaint of discrimination to a court.[116] Enforcing the non-discrimination obligation in the context of Ai systems escalates these challenges to exponential proportions. This is because of the opacity of these systems, affected persons often cannot detect instances of potential discrimination. And even where an individual suspects that unlawful discrimination may have occurred, limited access to the inner workings of the models or training data severely restricts their ability to meet the burden of proof requirements mandated by procedural law.

In an effort to address this challenge, Article 86 of the Act creates a new right to an explanation in the context of decisions taken on the basis of outputs of high-risk Ai systems. The right to an explanation for individual decision making is made necessary by the opacity and complexity of Ai systems. One of the major technical and legal challenges at the heart of Ai discourse is the inscrutable, opaque nature in which the algorithm functions. This is referred to as the “black box” problem, which obscures the intricate workings and decisional logic of complex Ai systems, making them “impenetrable,”[117] even as they make increasingly consequential decisions in society. Deep Learning algorithms take in millions of varied data points and correlates distinct data features to produce an output. Because this is primarily self-directed process, it presents substantial interpretive difficulties for data scientists, programmers, and end-users.[118] As Carstens and others put it, it leads to “decreased comprehensibility of the decisions and outcomes, their underlying criteria and reasons leading to certain decisions and of the weighting between those criteria. This results in a diminished ability of those affected to detect, prove and contest adverse outcomes such as manipulations or discriminations.”[119]

Moreover, algorithmic opacity is further occasioned by intellectual property protections coupled with the Act’s protective approach to confidential business information. If companies can shield their Ai models and algorithms behind confidentiality and intellectual property claims, it will become difficult for affected persons, regulators and the public to oversee or scrutinise these systems as required variously under the Act. As such, at Article 86(1) the Act provides that “any affected person” who has been subject to a “decision which is taken by the deployer on the basis of the output from a high-risk Ai system listed in Annex III” has “the right to obtain from the deployer clear and meaningful explanations of the role of the Ai system in the decision-making procedure and the main elements of the decision taken.”[120]

The right to an explanation is a welcomed step forward in improving transparency and accountability for high-risk Ai systems and facilitating the protection of fundamental rights. The Ai Act is more explicit than the GDPR in articulating the right to an explanation. The GDPR provides instead for a right to be informed about the logic involved in automated decision-making.[121] Specifically, Article 13(2)(f) requires controllers to provide “meaningful information about the logic involved” in automated decisions that have legal or significant effects on individuals.

The right to “meaningful information about the logic involved” begs the question; what does it mean, and what is required to explain an algorithm’s decision?[122] In the present context, the right plays a central role to accountability by empowering individuals to challenge adverse decisions made by Ai systems and seek legal remedies/redress where necessary. In this way, the right to an explanation acts as a prerequisite for effective judicial protection, because in theory it enables individuals to understand the basis of an Ai-based decision and potentially challenge it in a competent forum. However, some level of explainability in the Ai system is required for furnishing an explanation, this technical challenge presently a “roadblock” obstructing meaningful or effective explanations.[123]

The Act is silent, however, on the level of detail required for explanations under Article 86(1). It does not expect deployers to provide complex, detailed explanation on the functioning of the algorithm, in part because of a recognition that providing details in an explanation may prove challenging where the algorithm is opaque, because as Tran Viet Dung puts it, the “logic used may not be easy to describe and might not even be understandable in the first place.”[124] And the higher the level of autonomy, the more challenging it becomes to describe the decisional logic or processing activity.[125] The Ai Act mandates only that deployers provide “clear and meaningful explanations of the role of the Ai system in the decision-making procedure and the main elements of the decision taken.” It leaves open the exact form or content of these explanations, leaving significant room for interpretation and potential ambiguity.[126] This in turn will risk an inconsistent application of the right across different jurisdictions, Ai systems or contexts.

Further, what constitutes a “clear and meaningful” explanation must ordinarily vary depending on the complexity of the Ai system, the nature of the decision, and the affected person’s level of technical literacy and understanding. Similarly, there is a need for more guidance on the “main elements” of a decision. Consider for example the more specific, detailed language used at Article 13(2)(f) of the GDPR. It requires that data subjects be provided with “meaningful information about the logic involved” in automated decision-making, including “the significance and the envisaged consequences of such processing for the data subject.” This GDPR provision offers more specific guidance on the content of explanations relative to the Ai Act, unfortunately. Although the Commission is scheduled to come out with more guidance on the implementation of the Act, it is unclear whether this will include details on the standard of explanations under the Act.[127]

Although writing primarily in the context of Article 13 of the GDPR, the discourse on explainability has long been proposing specific and detailed facts that may form part of a meaningful explanation to give effect to the objective behind the obligation.[128] These include the confidence level or uncertainty associated with the Ai system’s output, the human oversight measures in place and how they influenced the final decision, the potential or known risks or limitations of the Ai system that could have affected the decision. There is clearly a need for more concrete regulatory guidance on what constitutes a “clear and meaningful” explanation and the “main elements” of a decision under the Ai Act.

Lastly, Article 86 is limited in its scope of application to Ai systems that have been classified as “high risk.” All other decision making or decision support systems, which are not classified as high risk are not covered by this provision, leaving open a wide range of decision support systems, which though not meeting the threshold of “high risk,” may nonetheless produce adverse legal effects or have similarly significant impacts on individuals’ health, safety, or fundamental rights.

2. Judicial powers to request and access any documentation under the Act

As made clear by Recital 170, the Act retains existing mechanism for redress under Union and national law in the event of algorithmic discrimination.[129] To that end, Recital 170 of the Preamble to the Act clarifies that “Union and national law already provide effective remedies to natural and legal persons whose rights and freedoms are adversely affected by the use of AI systems.” Accordingly, the handling algorithmic discrimination cases in the EU will still follows the traditional route for non-discrimination cases: In the first, a complainant or person affected by a discriminatory decision has the option to pursue an internal complaint with the respondent or respondent organisation.[130]

Failing this, affected persons may lodge formal legal proceedings with the relevant national equality body or a court with jurisdiction over discrimination claims.[131] To that end, Article 77 provides “national public authorities or bodies which supervise or enforce the respect of obligations under Union law protecting fundamental rights, including the right to non-discrimination” with “the power to request and access any documentation created or maintained under this Regulation in accessible language and format when access to that documentation is necessary for effectively fulfilling their mandates within the limits of their jurisdiction.”

Where such documentation proves insufficient to ascertain whether an infringement of obligations under Union law protecting fundamental rights has occurred, the public authority or body may make a reasoned request to the market surveillance authority, to organise testing of the high-risk Ai system through technical means.[132]

By making clear, as in Recital 170, that Union and national law already provide effective remedies for persons whose rights are adversely affected by Ai systems, and by clarifying in Recital 9 that the Act does not affect existing rights and remedies, the Act explicitly positions itself as not replacing or undermining the established legal avenues for challenging discrimination. This confirms that the primary legal framework for defining, prosecuting, and adjudicating discrimination, including algorithmic discrimination, remains with the existing body of Union and national non-discrimination law and the competent courts and equality bodies responsible for their enforcement. The traditional route for discrimination cases, involving internal complaints and formal legal proceedings with national authorities or courts, remains the principal path for seeking redress. The Act’s supporting role is particularly evident in how it enhances the effectiveness of these existing mechanisms, specifically through the provision in Article 77.

This is a welcomed measure, however, a significant challenge remains unaddressed in this context: the complainant bears the initial burden of proof to establish a prima facie case of discrimination.[133] This means that to success with an algorithmic discrimination claim, an individual or affected person must establishes firstly their membership in a protected class (e.g., race, gender, disability, etc.), secondly, that they were subjected to treatment that is less favourable than someone in a comparable situation who does not share the same protected characteristic. Finally, and more challenging, the complaint must prove that there exists a causal connection between the unfavourable treatment and their protected characteristic.[134]

While the exact elements of a prima facie case may vary across different jurisdictions, the requirement is generally that the complainant or person affected by a discriminatory decision must present evidence that establishes a reasonable inference of unlawful, unjustified discrimination – before a court will even admit the complaint or consider its merits.[135] The initial burden of establishing a prima facie case can be significant in traditional cases. However, in the context of algorithmic discrimination, this burden escalates exponentially, presenting new challenges that stand as significant barriers to establishing a prima facie case in the context of algorithmic discrimination, threatening to render remedies illusory for individuals who are not able to surmount these challenges.[136]

First is the increasing opacity and complexity of Ai Systems, particularly those using machine learning techniques to make or support decisions. The Act’s transparency measures seem to be premised on the optimistic assumption that the visible, transparent information will be both comprehensible and truthful. As noted earlier, complex decision systems often operate as “black boxes,” making it difficult and often impossible to understand to a discriminatory outcome, even for expert that have created the model. It obscures the reasoning behind Ai their decisions and the specific criteria they use or the specific factors that contribute, or their weighing. To the point off this contribution, this opacity also makes it difficult for affected persons to identify and challenge unfair or discriminatory outcomes. Effectively, algorithmic opacity obscures the causal link between the algorithm and the discriminatory effect, making it difficult of not impossible for the complainant to pinpoint the exact source of discrimination and provide sufficient evidence of this to meet the standard of a prima facie case.

Explainable Ai (XAi) is a specialised branch within the technical field of artificial intelligence, focuses on creating Ai systems whose actions and decisions can be easily understood and interpreted by humans, and systems that can otherwise help make other systems more explainable.[137] This field has been developing numerous technical bias definitions and fairness metrics, as well as practical techniques for bias detection and mitigation, with no lasting or scalable success.[138] As Deck et al. explain, formalisation and quantification cannot resolve fundamentally “normative issues – rooted in value conflicts.”[139] While these challenges can be supported by formal technical methods, it cannot entirely address the challenge.[140]

Secondly, proving algorithmic discrimination will rely on specialised types of evidence, including statistical or mathematical evidence and analysis demonstrating that the algorithmic outcomes disproportionately affect a particular group based on protected grounds such as race, gender, or age. To establish this, the complainant will likely need to access to confidential information such as the algorithm’s source code, decision-making processes, data inputs, expert evidence, internal company documents like logs or other technical documentation on the Ai system’s operations. This is considered proprietary information and may not be readily accessible to those alleging discrimination. While the Act does provide for an individual’s right to an explanation and for judicial access to documentation, it is unlikely that the level of detail and depth of any disclosure will include proprietary or confidential information. In fact, the act employs a necessarily protective approach confidentiality; I’ve written elsewhere about the interaction between the right to an explanation how the Act’s sweeping and totalising confidentiality obligations.

Third, the collecting and analysing this evidentiary data will be resource-intensive and will require specialised knowledge in statistics and data science – both on the part of the applicant, and on the part of the judicial officer who must make a decision. For the complainant, this means gathering and analysing large datasets, requiring the support of data scientists or statisticians. This process will be resource intensive and time-consuming, potentially resulting in a de facto exclusion of individuals who lack the means to undertake such a complex endeavour. The financial burden of legal representation, already a significant hurdle for many individuals, is likely to escalate exponentially in the context of algorithmic discrimination claims; regrettably, the Act is unresponsive to this reality. Furthermore, the judicial officer tasked with deciding the case must also possess the technical knowledge to interpret the statistical evidence.[141] The specialised and resource-intensive nature of this process in the context of Ai discrimination creates a significant disadvantage for individuals affected by Ai discrimination, potentially rendering the right to non-discrimination illusory for many.

The effectiveness of all legal mechanisms for successfully launching discrimination claims centre on the complainant’s ability to provide sufficient evidence to establish a prima facie case of discrimination based on prohibited grounds. In the context of algorithms, this burden escalates exponentially, presenting significant limits to the accessibility and effectiveness of existing legal redress mechanisms, potentially undermining the very right to non-discrimination.

Conclusion

The European Union’s Artificial Intelligence Act constitutes a significant regulatory response to the risks posed by biased Ai systems. However, as this analysis demonstrates, the Act adopts a predominantly technical approach to bias, positioning itself in a crucial, yet ultimately supporting role relative to established Union non-discrimination frameworks. The Act’s strategy, centred on high-risk Ai systems, mandates technical measures to identify and mitigate biases, notably under Article 10 concerning data governance. Recognizing that biases can be inherent in data sets, particularly historical ones, or generated in real-world settings,[142] the Act requires training, validation, and testing data to meet quality criteria, including the examination for and correction of biases likely to lead to discrimination prohibited under Union law.[143] Article 10(5) supports this by permitting the processing of special categories of personal data when strictly necessary for bias detection and correction.[144] This technical focus provides developers with tools to build systems less likely to perpetuate data biases.

Crucially, this technical approach operates distinctly from the normative, outcome-oriented nature of legal non-discrimination. As Meding’s analysis highlights, technical fairness and legal non-discrimination, while sharing goals, differ conceptually and methodologically.[145] The Act addresses the technical cause of potential bias in data and algorithms, while Union non-discrimination law governs the legal effect of unjust differential treatment based on prohibited grounds.[146] Consequently, the Ai Act functions as an essential, Ai-specific layer of preventative regulation that supports existing non-discrimination law. By imposing technical obligations like those in Article 10, the Act helps duty-holders reduce the likelihood of discriminatory outcomes, operationalising non-discrimination principles during Ai design and development.

Nonetheless, this supportive function has inherent limits; compliance with Article 10 and other technical mandates cannot guarantee the elimination of discrimination as required by law. An Ai system can technically comply with the Act and still result in discrimination upon deployment, as discrimination can arise from factors beyond initial bias mitigation, including real-world context, dynamic proxies, and how outputs are used. Furthermore, a critical perspective highlights that the Act is not primarily a human rights document, but rather a market regulation focused on product safety and compliance bureaucracy for Ai developers.[147] Criticisms suggest it may lead to superficial compliance rather than ensuring real accountability for fundamental rights risks,[148] being seen as a regulation for companies, not people, partly due to minimal direct user obligations concerning those affected by Ai systems.[149] Reports from NGOs criticise the Act for not being a robust human rights document, citing a lack of robust redress mechanisms and exemptions, illustrating it is not fundamentally designed as a human rights document.[150] They argue the focus on promoting Ai uptake potentially comes at the expense of safeguarding rights.[151]

However, seen from this perspective, the Act doesn’t need to be a human rights document in the primary sense. Its purpose is to create a harmonised internal market for trustworthy Ai systems, ensuring safety and managing specific Ai-related risks within a framework that complements, rather than replaces, existing human rights and non-discrimination law. The Act integrates fundamental rights safeguards as essential requirements for market access and deployment, without transforming into a standalone human rights treaty or a primary redress mechanism.

This understanding of the Act as primarily a market and safety regulation with integrated safeguards reinforces its supporting role in the governance of non-discrimination. If its foundational purpose is regulating Ai products for the market, its contribution to non-discrimination is naturally facilitative. Its focus on technical compliance aligns with supporting existing legal frameworks by addressing Ai-specific technical vulnerabilities. Critiques regarding the lack of robust direct redress within the Act also align with the idea that primary enforcement remains with established human rights and non-discrimination legal systems.

[1] I employ a lowercase ‘i’ in the term ‘artificial intelligence’ throughout this paper to signify that I do not regard these systems as ‘intelligent’. This typographical choice to use a lowercase ‘i’ in ‘intelligence’ underscores this perspective.

[2] Maslej, Nestor and others, ‘The AI Index 2023 Annual Report’ (AI Index Steering Committee, 4 April 2023) https://aiindex.stanford.edu/wp-content/uploads/2023/04/HAI_AI-Index-Report_2023.pdf accessed 4 May 2025. See also: Zhang, Kirsty and others, ‘The AI Index 2024 Annual Report’ (AI Index Steering Committee, May 2024) https://arxiv.org/abs/2405.19522.

[3] Dass, Kumar and others, ‘Detecting Racial Inequalities in Criminal Justice: Towards an Equitable Deep Learning Approach for Generating and Interpreting Racial Categories Using Mugshots’ (2023) AI & SOCIETY 897.

[4]  Pan and others, ‘The Adoption of Artificial Intelligence in Employee Recruitment: The Influence of Contextual Factors’ (2022) Int J Hum Resour Manag 1125; Köchling and Wehner, ‘Discriminated by an Algorithm: A Systematic Review of Discrimination and Fairness by Algorithmic Decision-Making in the Context of HR Recruitment and HR Development’ (2020) Bus Res 795, wherein on the basis of a systematic review of 36 journal articles from 2014 to 2020, the authors review discrimination by algorithmic decision-making in the human resource management context.

[5] Cao, ‘AI in Finance: Challenges, Techniques, and Opportunities’ (2022) ACM Computing Surveys 64:1.

[6] Kosta, Algorithmic state surveillance: Challenging the notion of agency in human rights  (2022) Regulation & Governance, 16 (212-224).

[7] Joh, The New Surveillance Discretion: Automated Suspicion, Big Data, and Policing, (2016) 10 HARV. L. & POL’Y REV. 15; Citron & Pasquale, The Scored Society: Due Process for Automated Predictions, WASH L REV (2014) 1 (89).

[8] Crompton and Diane Burke, ‘Artificial Intelligence in Higher Education: The State of the Field’ (2023) 20 International Journal of Educational Technology in Higher Education 22; Pham and Sampson, ‘The Development of Artificial Intelligence in Education: A Review in Context’ (2022) Journal of Computer Assisted Learning 38 (1408).

[9] Surber, ‘Artificial Intelligence: Autonomous Technology (AT), Lethal Autonomous Weapons Systems (LAWS) and Peace Time Threats’ Artificial Intelligence, (2018) ICT for Peace, https://ict4peace.org/wp-content/uploads/2018/02/2018_RSurber_AI-AT-LAWS-Peace-Time-Threats_final.pdf 

[10] Yamin and others, ‘Weaponized AI for Cyber Attacks’ (2021) Journal of Information Security and Applications 57.

[11] Marwala, ‘Closing the Gap: The Fourth Industrial Revolution in Africa’ October (2020) Pan Macmillan South Africa; Adams, Pienaar, Olorunju, Gaffley, Gastrow, Thipanyane, Ramkissoon, Van der Berg & Adams, ‘Human rights and the fourth industrial revolution in South Africa’ (2021) Cape Town: HSRC Press; Cozgarea, Cozgarea & Stanciu, ‘Artificial Intelligence Applications In The Financial Sector,’ (2008) Theoretical and Applied Economics, Editura Economica, vol. 12 (57-62).

[12] Solove, Artificial Intelligence and Privacy (2024) Florida Law Review 1 (77).

[13] Barocas and Selbst, ‘Big Data’s Disparate Impact’ (2016) California Law Review 104 (671-732).

[14] Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts [2024] OJ L 252/1.

[15] Ibid; Anupam Datta and others, ‘Proxy Non-Discrimination in Data-Driven Systems’(2017) arXiv.org; Johnson, ‘Algorithmic Bias: On the Implicit Biases of Social Technology’ Synthese 198 (10).

[16] Johnson, (2021) Synthese 198 (10).

[17] Johnson, (2021) Synthese 198 (10), discussing the trade off with accuracy in algorithmic decisions and predictions.

[18] Johnson, (2021) Synthese 198 (10).

[19] Deck and others, ‘Implications of the AI Act for Non-Discrimination Law and Algorithmic Fairness’ (2024) arXiv.org

[20] Coglianese & Lehr, Transparency and Algorithmic Governance (2019) 71 ADMIN. L. REV 1 (15) noting that ‘The algorithm itself tries many possible combinations of variables, figuring out how to put them together to optimize the objective function.’

[21] Xie, ‘An explanation of the relationship between artificial intelligence and human beings from the perspective of consciousness’ (2021) Cultures of Science 4(3) 124-134.

[22] Ibid.

[23] Datta and others, (2017) arXiv.org; Prince, and Schwarcz, ‘Proxy Discrimination in the Age of Artificial Intelligence and Big Data’ (2020) 105 Iowa Law Review 1257; Patty and Penn, ‘Algorithmic Fairness and Statistical Discrimination’ (2022) Philosophy Compass e12891; Johnson, (2021) Synthese 198 (11).

[24] For instance, human minds often infer emotions from facial expressions, use money as a proxy for value, or use job titles as proxy for expertise; See Kemp and Tenenbaum, ‘Structured Statistical Models of Inductive Reasoning’ (2009) Psychological Review 116 (20);  Johnson, (2021) Synthese 198 (10).

[25] Johnson, (2021) Synthese 198 (10).

[26] Goodman and Flaxman, ‘European Union Regulations on Algorithmic Decision-Making and a “Right to Explanation” (2017) AI Magazine 38.

[27] Pouget and Zuhdi, ‘AI and Product Safety Standards Under the EU AI Act’ (2024) Carnegie Endowment.

[28] This definition aligns with the definition of risk in other NLF legislation and e.g. with Safety Risk Management per ISO Guide 51.

[29] Martens, ‘The European Union AI Act: Premature or Precocious Regulation?’ (Bruegel | The Brussels-based economic think tank, 23 May 2024) <https://www.bruegel.org/analysis/european-union-ai-act-premature-or-precocious-regulation> accessed 30 June 2024.

[30] See Article 10(2)(f) of the Ai Act.

[31] Meding, ‘It’s complicated. The relationship of algorithmic fairness and non-discrimination regulations in the EU AI Act’ (2025) arXiv.org 2501.12962v2.

[32] Barocas, Hardt & Narayanan, Fairness and Machine Learning: Limitations and Opportunities (2023) MIT Press; Rodolfa, Lamba & Ghani, ‘Empirical Observation of Negligible Fairness–Accuracy Trade-Offs in Machine Learning for Public Policy’ (2021) Nature Machine Intelligence (10) 896; Selbst et al., ‘Fairness and Abstraction in Sociotechnical Systems’ (2019) Conference on Fairness, Accountability, and Transparency 59.

[33] Burden v. The United Kingdom App no 13378/05 (ECtHR [GC], 29 April 2008), para. 60; decisions of the ECtHR can be accessed via https://hudoc.echr.coe.int/eng with their case number or party names; Guberina v. Croatia App no 23682/13 (ECtHR, 22 March 2016), para. 69; D.H. and Others v. the Czech Republic App no 57325/00 (ECtHR [GC], ECHR 2007-IV), para. 175.

[34] Belgium’ Linguistics v Belgium App nos 1474/62, 1677/62, 1691/62, 1769/63, 1994/63, 2126/64 (ECtHR, 23 July 1968)

[35] This is evidenced by a heightened focus on data quality, (see Article 10 generally). See also Recital 67, which highlights that biases can be inherent in underlying data sets and emphasise the importance of high-quality data sets for training Ai systems, particularly to avoid perpetuating and amplifying existing discrimination.

[36] See for example Article 9 of the Act which requires a risk management system for high-risk Ai systems, which includes identifying and mitigating risks related to the Ai system’s design. While Article 15 requires that high-risk AI systems be designed and developed in a way that ensures their accuracy, robustness, and cybersecurity.

[37] To that end, Article 10(4) requires that data sets take into account, the characteristics or elements that are particular to the specific geographical, contextual, behavioural or functional setting within which the high-risk Ai system is intended to be used. Accordingly, Article 14 requires appropriate human oversight of high-risk Ai systems. Further, Recital 13 introduces the concept of “reasonably foreseeable misuse,” in recognition of the fact that the way an Ai system is deployed and used can also lead to discriminatory outcomes. Recital 86 also emphasises the need for deployers to understand the context of use and identify potential risks not foreseen in the development phase.

[38] Wachter et al., ‘Why fairness cannot be automated: Bridging the gap between EU non-discrimination law and AI’ (2021) Computer Law & Security Review 105567; Nishant et al, ‘The Formal Rationality of Artificial Intelligence-based Algorithms’ (2024) Journal of Information Technology 39 (20); Lindebaum et al., ‘Insights From ‘The Machine Stops” to Better Understand Rational Assumptions in Algorithmic Decision Making and Its Implications for Organisations’ (2020) Academy of Management Review 45 (247); Selbst et al., ‘Fairness and Abstraction in Sociotechnical Systems’ in Conference on Fairness, Accountability, and Transparency’ (2019) 59; Matwin etal., ‘Algorithmic Decision Making and the Cost of Fairness’ International Conference on Knowledge Discovery and Data Mining (2017) 797.

[39] Wachter et al., (2021) Computer Law & Security Review.

[40] Wachter et al., (2021) Computer Law & Security Review; Nishant et al. (2024) Journal of Information Technology 39 (21);

[41] Like those in Article 10, including the ability to process sensitive data under strict conditions via Article 10(5)

[42] These are design requirements for high-risk Ai systems; actor specific obligations; including obligations promoting transparency and explainability. This interpretation is shared by Lukas Arnold, ‘How the European Union’s AI Act Provides Insufficient Protection Against Police Discrimination’ (2024) University of Pennsylvania Carey Law School.

[43] At para 10 of the Preamble to the Act, it makes clear that it “does not seek to affect the application of existing Union law governing the processing of personal data. See also para 45, 48, 67, 70 of Preamble.

[44] At para 45 of the Preamble to the Act makes clear that “Practices that are prohibited by Union law, including … non- discrimination law, … should not be affected by this Regulation. While paragraphs 29 (based on guiding principles of the 2019 Ethics guidelines for trustworthy AI developed by the independent AI HLEG) understands “Diversity, non-discrimination and fairness” as meaning “AI systems are developed and used in a way that includes diverse actors and promotes equal access, gender equality and cultural diversity, while avoiding discriminatory impacts and unfair biases that are prohibited by Union or national law. The opening line of para 67 of the Preamble underscores the importance of High-quality data in ensuring that “high-risk AI system performs as intended and safely and it does not become a source of discrimination prohibited by Union law.” It goes on to confirms that “The data sets should also have the appropriate statistical properties… with specific attention to the mitigation of possible biases in the data sets, that are likely to … lead to discrimination prohibited under Union law.”

[45] See also Recital 7, that the AI Act aims to complement existing Union law, including legislation on fundamental rights and non-discrimination.

[46] Finally, and to the extent that this reliance on existing union non-discrimination law is unclear, the Act at Article 96(1)(e) creates an obligation for the Commission to develop guidelines on its practical application, including “detailed information on the relationship of this Regulation with … relevant Union law.

[47] Convention for the Protection of Human Rights and Fundamental Freedoms (European Convention on Human Rights, as amended) (ECHR), which provides that “[t]he enjoyment of the rights and freedoms set forth in this Convention shall be secured without discrimination on any ground such as sex, race, colour, language, religion, political or other opinion, national or social origin, association with a national minority, property, birth or other status.”

[48] Charter of Fundamental Rights of the European Union [2012] OJ C 326/391, which provides that “[a]ny discrimination based on any ground such as sex, race, colour, ethnic or social origin, genetic features, language, religion or belief, political or any other opinion, membership of a national minority, property, birth, disability, age or sexual orientation shall be prohibited.”

[49] Council Directive 2000/43/EC of 29 June 2000 implementing the principle of equal treatment between persons irrespective of racial or ethnic origin [2000] OJ L 180/22, which provides that “the principle of equal treatment shall mean that there shall be no direct or indirect discrimination based on racial or ethnic origin.”

[50] Council Directive 2000/78/EC of 27 November 2000 establishing a general framework for equal treatment in employment and occupation [2000] OJ L 303/16, which “lay[s] down a general framework for combating discrimination on the grounds of religion or belief, disability, age or sexual orientation as regards employment and occupation.”

[51] Directive 2006/54/EC of the European Parliament and of the Council of 5 July 2006 on the implementation of the principle of equal opportunities and equal treatment of men and women in matters of employment and occupation (recast) [2006] OJ L 204/23, which provides that “For the same work or for work to which equal value is attributed, direct and indirect discrimination on grounds of sex with regard to all aspects and conditions of remuneration shall be eliminated.”

[52] Council Directive 2004/113/EC of 13 December 2004 implementing the principle of equal treatment between men and women in the access to and supply of goods and services [2004] OJ L 373/37, which provides that “the principle of equal treatment between men and women shall mean that (a) there shall be no direct discrimination based on sex, including less favourable treatment of women for reasons of pregnancy and maternity; (b) there shall be no indirect discrimination based on sex.”

[53] Council Directive on implementing the principle of equal treatment between persons irrespective of religion or belief, disability, age or sexualorientation {SEC (2008) 2180, which provides a framework for “combating discrimination on the grounds of religion or belief, disability, age, or sexual orientation”.

[54] Gerards & Borgesius, ‘Protected Grounds and the System of Non-Discrimination Law in the Context of Algorithmic Decision-Making and Artificial Intelligence’ (2021) Colo. Tech. L.J 55.

[55] To the extent that the High-Risk Ai system does not rely on data and Machine Learning techniques, paragraphs 2 to 5 apply only to the testing data sets.

[56] Article 10(2)(a)-(h) provides that “[t]raining, validation and testing data sets shall be subject to data governance and management practices appropriate for the intended purpose of the high-risk AI system. Those practices shall concern in particular: (a) the relevant design choices; (b) data collection processes and the origin of data, and in the case of personal data, the original purpose of the data collection; (c) relevant data-preparation processing operations, such as annotation, labelling, cleaning, updating, enrichment and aggregation; (d) the formulation of assumptions, in particular with respect to the information that the data are supposed to measure and represent; (e) an assessment of the availability, quantity and suitability of the data sets that are needed; (f) examination in view of possible biases that are likely to affect the health and safety of persons, have a negative impact on fundamental rights or lead to discrimination prohibited under Union law, especially where data outputs influence inputs for future operations; (g) appropriate measures to detect, prevent and mitigate possible biases identified according to point (f); (h) the identification of relevant data gaps or shortcomings that prevent compliance with this Regulation, and how those gaps and shortcomings can be addressed.” Own Emphasis.

[57] That is to say, it is not an obligation of result, rather, its obligation of conduct.

[58] Article 10(3) of the Act.

[59] Article 10(4) of the Act.

[60] Wortham, ‘Garbage in, toxic data out: a proposal for ethical artificial intelligence sustainability impact statements’ (2023) AI and Ethics 3 (135–1142); Geiger and others, ‘Garbage in, Garbage out’ Revisited: What Do Machine Learning Application Papers Report about Human-Labelled Training Data?’ (2021) Quantitative Science Studies 2 (795).

[61] Art 10(2) Artificial Intelligence Act.

[62] Meding, arXiv.org, 10.

[63] Meding, arXiv.org, 10.

[64] Ibid. This supported by reference to translational consistency in the German translation.

[65] Ibid.

[66] Ibid.

[67] Ibid.

[68] Ibid.

[69] Ibid.

[70] Ibid.

[71] Broussard, ;Artificial Unintelligence: How Computers Misunderstand the World’ (2018) The MIT Press 103.

[72] Delbosc, ‘There Is No Such Thing as Unbiased Research – Is There Anything We Can Do about That?’ (2023) Transport Reviews 33 (155); Maatman, ‘Unbiased Machine Learning Does Not Exist (LBBOnline’ October 2018) Available at <https://www.lbbonline.com/news/unbiased-machine-learning-does-not-exist-3> accessed 19 October 2023; Johnson, (2021) Synthese 198 (10); du Preez, ‘AI and Ethics – “Unbiased Data Is an Oxymoron’ (31 October 2019) Available at <https://diginomica.com/ai-and-ethics-unbiased-data-oxymoron> accessed 19 October 2023.

[73] Bower, ‘The Nature of Data and Their Collection’, in Bower (ed.), Statistical Methods for Food Science: Introductory Procedures for the Food Practitioner, 2nd edn. (Hoboken, 2013) 15.

[74] Eaglin, ‘Constructing Recidivism Risk’ (2017) Emory Law Journal 67 (59), as discussed and quoted in Solove (2024).

[75] Burk, ‘Algorithmic Legal Metrics’ (2020) Notre Dame Law Review 96 (1147).

[76] Solove, The Digital Person: Technology and Privacy in the Information Age (New York, 2004) 3.

[77] Seijo-Pardo and others, ‘Biases in Feature Selection with Missing Data’ (2019) Neurocomputing 432 (97).

[78] O’Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (2016) Crown Publishing Group.

[79] Mehrabi and others, ‘A Survey on Bias and Fairness in Machine Learning’ (2021) ACM Computing Surveys 54 (115); Samuel, ‘Why It’s so Damn Hard to Make AI Fair and Unbiased’ (Vox, 19 April 2022) <https://www.vox.com/future-perfect/22916602/ai-bias-fairness-tradeoffs-artificial-intelligence> accessed 19 October 2023.

[80] Barocas and Selbst, (2016) California Law Review 671.

[81] Datta and others, (2017) arXiv.org.

[82] Gillis, ‘The Input Fallacy’ (2022) Minnesota Law Review 106 (1175).

[83] Chander, ‘EU’s AI Law Needs Major Changes to Prevent Discrimination and Mass Surveillance – European Digital Rights (EDRi)’ <https://edri.org/our-work/eus-ai-law-needs-major-changes-to-prevent-discrimination-and-mass-surveillance/> accessed 29 May 2024.

[84] Article 10(3) of the Ai Act.

[85] While the emphasis in Article 10 is heavily on data inputs and governance, it’s important to note that the Act for high-risk systems also includes requirements related to Risk Management System, Quality Management System, and Post-Market Monitoring. Discriminatory outcomes, particularly those impacting fundamental rights, could be monitored under these general provisions. Notably, however, the Act’s provisions on output monitoring and real-world impact assessment are less detailed and prescriptive compared to the requirements for input data and development processes.

[86] Quezada-Tavarez, Dutkiewicz and Krack, ‘Voicing Challenges: GDPR and AI Research’ (2022) Open research Europe 2 (126); Demircan, Kalyna, ‘Europe: The EU AI Act’s Relationship with Data Protection Law: Key Takeaways’ Privacy Matters, April 2024) <https://privacymatters.dlapiper.com/2024/04/europe-the-eu-ai-acts-relationship-with-data-protection-law-key-takeaways/> accessed 1 July 2024.

[87] See Article 9 of GDPR.

[88] Lukas Feiler and others, ‘EU AI Act: Diversity and Inclusion Prevails over Data Protection’ (Lexology, 26 June 2024), 3, <https://www.lexology.com/library/detail.aspx?g=a978bb5a-409a-4b26-8df1-26e3244bd97f> accessed 29 June 2024. As they write “On the one hand, ignoring personal demographic data promotes the same risk as the widely rejected idea of fairness through unawareness because legally protected attributes like race and gender usually correlate to innocuous proxy variables. If protected attributes are unavailable during model training and evaluation, these subtle correlations cannot be accounted for, nor can technical fairness metrics be tested and optimized.”

[89] Yucer and others, ‘Measuring Hidden Bias within Face Recognition via Racial Phenotypes’ (2021) EEE Winter Conference on Applications of Computer Vision, WACV, 2022.

[90] Artzt and Dung, Artificial Intelligence and Data Protection: How to Reconcile Both Areas from the European Law Perspective’ (2023) Vietnamese Journal of Legal Sciences 7(2):39-58; Paterson and McDonagh, ‘Data protection in an era of big data: the challenges posed by big personal data’ (2019) Monash University.

[91] Deck et al., (2023) arXiv 3.

[92] Ibid, as they note, “discrimination and fairness considerations can provide a justification for data processing during the training phase of high-risk AI systems.”

[93] Marvin van Bekkum and Frederik Zuiderveen Borgesius, ‘Using Sensitive Data to Prevent Discrimination by Artificial Intelligence: Does the GDPR Need a New Exception?’ (2023) 48 Computer Law & Security Review 48 (105770).

[94] IBM AI Fairness 360 is an open-source toolkit with algorithms for detecting and mitigating bias in machine learning models.

[95] A visualization tool to explore how machine learning models behave with different data inputs and help identify biases: https://pair-code.github.io/what-if-tool/

[96] Microsoft’s Fairlearn is an open-source toolkit to assess and improve the fairness of AI systems: https://fairlearn.org/

[97] While the GDPR provides the general framework for data protection, Article 10(5) acts as a lex specialis for the processing of special categories of personal data in the context of Ai. This means that the specific conditions and safeguards of Article 10(5) take precedence over the general provisions of the GDPR in this particular context.

[98] Article 10(5)(a)-(f).

[99] Article 10(5)(a)-(f).

[100] Article 10(5)(a)-(f): “(a) the bias detection and correction cannot be effectively fulfilled by processing other data, including synthetic or anonymised data; (b) the special categories of personal data are subject to technical limitations on the re-use of the personal data, and state of the art security and privacy-preserving measures, including pseudonymisation; (c) the special categories of personal data are subject to measures to ensure that the personal data processed are secured, protected, subject to suitable safeguards, including strict controls and documentation of the access, to avoid misuse and ensure that only authorised persons with appropriate confidentiality obligations have access to those personal data; (d) the personal data in the special categories of personal data are not to be transmitted, transferred or otherwise accessed by other parties;(e) the personal data in the special categories of personal data are deleted once the bias has been corrected or the personal data has reached the end of its retention period, whichever comes first; (f) the records of processing activities pursuant to Regulations (EU) 2016/679 and (EU) 2018/1725 and Directive (EU) 2016/680 include the reasons why the processing of special categories of personal data was strictly necessary to detect and correct biases, and why that objective could not be achieved by processing other data.”

[101] Solove, (2024) Florida Law Review 1 (77); Staab, Vero, Balunović and Vechev, ‘Beyond Memorization: Violating Privacy Via Inference with Large Language Models’ (arXiv, 6 May 2024) https://doi.org/10.48550/arXiv.2310.07298 accessed 4 May 2025 noting that “It is often technically very difficult to separate personal data from non-personal data.”

[102] Moira Paterson and Maeve McDonagh, ‘Data Protection in an Era of Big Data: The Challenges Posed by Big Personal Data’ (2018) Monash University law review 1 (44).

[103] Ai is currently very expensive to train, especially models that would be defined as high risk. See David Meyer, ‘The Cost of Training AI Could Soon Become Too Much to Bear | Fortune’ <https://fortune.com/2024/04/04/ai-training-costs-how-much-is-too-much-openai-gpt-anthropic-microsoft/> ; Derek du Preez, ‘AI Is Currently Too Expensive to Take Most of Our Jobs, Finds MIT Researchers’ (24 January 2024) <https://diginomica.com/ai-currently-too-expensive-take-most-our-jobs-finds-mit-researchers.

[104] For a lack of a suitable legal basis for the collection and processing of personal data for the purpose of training the algorithms underlying ChatGPT. See Natasha Lomas, ‘ChatGPT Is Violating Europe’s Privacy Laws, Italian DPA Tells OpenAI’ (TechCrunch, 29 January 2024) <https://techcrunch.com/2024/01/29/chatgpt-italy-gdpr-notification/>.

[105] ‘Italian DPA Fines Food Delivery App 2.6M Euros for GDPR Violations’ <https://iapp.org/news/b/italian-dpa-fines-food-delivery-app-3m-euros-for-gdpr-violations> accessed 1 July 2024.

[106] ‘The French SA Fines Clearview AI EUR 20 Million | European Data Protection Board’ <https://www.edpb.europa.eu/news/national-news/2022/french-sa-fines-clearview-ai-eur-20-million_en> accessed 1 July 2024; Kettas (n 104).

[107]  Staab and others, ‘Beyond Memorization’ (arXiv, 6 May 2024); Solove, (2024) Florida Law Review 1 (77).

[108] Stefanie James and others, ‘Synthetic Data Use: Exploring Use Cases to Optimise Data Utility’ (2021) Discover Artificial Intelligence 1 (15); Abdul Majeed and Sungchang Lee, ‘Anonymization Techniques for Privacy Preserving Data Publishing: A Comprehensive Survey’ (2021) IEEE Access 8512, highlighting re-identification methods used by malevolent adversaries to re-identify people uniquely from the privacy preserved data.

[109] Rainer Mühlhoff and Hannah Ruschemeier, ‘Updating Purpose Limitation for AI: A Normative Approach from Law and Philosophy’ (2024) International Journal of Law and Information Technology.

[110] Boris P Paal, ‘Artificial Intelligence as a Challenge for Data Protection Law: And Vice Versa’ in Oliver Mueller and others (eds), The Cambridge Handbook of Responsible Artificial Intelligence: Interdisciplinary Perspectives (Cambridge University Press 2022); Moira Paterson and Maeve (2018) Monash University law review 1 (44); Matthias Artzt and TV Dung, ‘Artificial Intelligence and Data Protection: How to Reconcile Both Areas from the European Law Perspective’ (2022) Vietnamese Journal of Legal Sciences.

[111] Artur Varanda and others, ‘Log Pseudonymization: Privacy Maintenance in Practice’ (2021) 63 Journal of Information Security and Applications 103021.

[112] Paul Boudolf, Imagery Pseudonymization: Using Ai For Privacy Enhancement (2020) UniGent.

[113] Rishav Chourasia and Neil Shah, ‘Forget Unlearning: Towards True Data-Deletion in Machine Learning’, Proceedings of the 40th International Conference on Machine Learning (PMLR 2023) <https://proceedings.mlr.press/v202/chourasia23a.html> accessed 30 June 2024; See also Zachary Izzo and others, ‘Approximate Data Deletion from Machine Learning Models’, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (PMLR 2021) <https://proceedings.mlr.press/v130/izzo21a.html> accessed 30 June 2024.

[114] Lukas Arnold, (2024) University of Pennsylvania Carey Law School.

[115] Deck et al., (2024) arxiv.org; See for example Laina R.R., ‘Proving an Employer’s Intent: Disparate Treatment Discrimination and the Stray Remarks Doctrine after Reeves v. Sanderson Plumbing Products‘ (2002) Vanderbilt Law Review 55 (219, 219-259). Ponce (2022) Computer Law and Security Review; Fredman, Discrimination Law, 2nd edn. (Oxford, 2011) 45.

[116] Ponce, ‘Direct and Indirect Discrimination Applied to Algorithmic Systems: Reflections to Brazil’ (2022) Computer Law and Security Review.

[117] Matthias Artzt and TV Dung, (2022) Vietnamese Journal of Legal Sciences.

[118] Georgios Pavlidis, ‘Unlocking the Black Box: Analysing the EU Artificial Intelligence Act’s Framework for Explainability in AI’ (2024) Law, Innovation and Technology.

[119] Dr Carsten Orwat, ‘Risks of Discrimination through the Use of Algorithms: A study compiled with a grant from the Federal Anti-Discrimination Agency (2019) Germany Federal Anti-Discrimination Agency.

[120] The list of high-risk Ai systems at Annex III is sufficiently broad to cover contexts where Ai decision systems might result in discrimination; it includes are those in administration of justice and democratic processes, migration, asylum and border control management; law enforcement; access to and enjoyment of essential private services and essential public services and benefits; employment, workers management and access to self-employment; education and vocational training; and Biometrics – to the extent that their uses are permitted under relevant Union or national law.

[121] This right is established in Articles 13-15 and 22 of the GDPR.

[122] Bryce Goodman and Seth Flaxman, ‘European Union Regulations on Algorithmic Decision-Making and a “Right to Explanation”’ (2024) Ai Magazine.

[123] Matthias Artzt and TV Dung, (2022) Vietnamese Journal of Legal Sciences.

[124] ibid.

[125] Ibid.

[126] Gagandeep Kaur, ‘Concerns Remain Even as the EU Reaches a Landmark Deal to Govern AI – ProQuest’ <https://www.proquest.com/docview/2900586403?parentSessionId=blMDkbwsWWQ%2BD7Gb3Oe5k1SQ5d9XcV3czoWfJyYVLSU%3D&pq-origsite=primo&accountid=14682&sourcetype=Trade%20Journals> accessed 29 June 2024.

[127] See Article 73, 6 and 96 of the Act.

[128] Fink, & Finck, ‘Reasoned A(I)dministration: explanation requirements in EU law and the automation of public administration’ (2022) European Law Review 47(376-392). Retrieved from https://hdl.handle.net/1887/3439725; Cecilia Panigutti and others, ‘The Role of Explainable AI in the Context of the AI Act’ (2023) Conference on Fairness, Accountability and Transparency; Georgios Pavlidis, ‘Unlocking the Black Box: Analysing the EU Artificial Intelligence Act’s Framework for Explainability in AI’ (2024) Law, Innovation and Technology.

[129] See Recital 170 of the Preamble read with Article 77 of the Ai Act.

[130] In addition to this, individuals can also lodge complaints with national supervisory authorities, designated in each member State to oversee the Act’s implementation in terms of Article 85 and Recital 170 of the EU Ai Act. These authorities are empowered to investigate, audit, and enforce corrective administrative measures; See Article 85, 79, 74, and 70 of the EU Ai Act.

[131] Recital 9 of the Preamble to the Act clarifies that the Act does not affect existing rights and remedies under Union law. This confirms that affected persons are still required to rely on existing legal frameworks and institutions, including judicial remedies, to address discrimination caused by Ai systems.

[132] See Article 77(3) of the EU Ai Act.

[133] Ross, ‘The Burden of Proving Discrimination’ (2000) International Journal of Discrimination and the Law 4(2).

[134] Borgesius, ‘Discrimination, Artificial Intelligence, and Algorithmic Decision-Making’ (2018) Strasbourg: Council of Europe <https://dare.uva.nl/search?identifier=7bdabff5-c1d9-484f-81f2-e469e03e2360> accessed 30 June 2024.

[135] ibid.

[136] Jessica Lind, ‘The Prima Facie Case of Age Discrimination in Reduction-in-Force Cases’ (1995) Michigan Law Review 94 (832); A Fedorchuk, ‘PROOVING IN CASES OF DISCRIMINATION IN THE FIELD OF LABOUR’ (2020) Bulletin of Taras Shevchenko National University of Kyiv. Legal Studies 59.

[137] Cecilia Panigutti et al. ‘The role of explainable AI in the context of the AI Act’ (2023) FAccT ’23: the 2023 ACM Conference on Fairness, Accountability, and Transparency

[138] Friedler, Scheidegger and Venkatasubramanian, ‘The (Im)Possibility of Fairness’ (2021) Communications of the ACM 136; Creel and Hellman, ‘The Algorithmic Leviathan: Arbitrariness, Fairness, and Opportunity in Algorithmic Decision-Making Systems’ (2022) Canadian Journal of Philosophy 1.

[139] Deck and others, ‘Implications of the AI Act for Non-Discrimination Law and Algorithmic Fairness’ (2024) arXiv.org

[140] ibid.

[141] Article 26(2) of the Act mandates that deployers of high-risk Ai systems assign human oversight to individuals with the “necessary competence, training, and authority.” While not explicitly mentioning judges, this provision implies that those overseeing AI systems in the judicial context should have adequate training.

[142] Artificial Intelligence Act, Recital 67.

[143] Article 10(2)(f).

[144] Article 10(5).

[145] Meding, arXiv.org, 10.

[146] See e.g. Council Directive 2000/43/EC implementing the principle of equal treatment between persons irrespective of racial or ethnic origin [2000] OJ L 180/22; Council Directive 2000/78/EC establishing a general framework for equal treatment in employment and occupation [2000] OJ L 303/16.

[147] Chander, ‘EU’s AI Law Needs Major Changes to Prevent Discrimination and Mass Surveillance – European Digital Rights (EDRi)’ <https://edri.org/our-work/eus-ai-law-needs-major-changes-to-prevent-discrimination-and-mass-surveillance/> accessed 29 May 2024; Coalition of Digital, Human Rights and Social Justice Groups, ‘EU’s AI Act fails to set gold standard for human rights’ (Joint Statement/Analysis, 3 April 2024), https://www.amnesty.eu/news/eus-ai-act-fails-to-set-gold-standard-for-human-rights/

[148] Ibid.

[149] Chander, ‘EU’s AI Law Needs Major Changes to Prevent Discrimination and Mass Surveillance – European Digital Rights (EDRi)’ <https://edri.org/our-work/eus-ai-law-needs-major-changes-to-prevent-discrimination-and-mass-surveillance/> accessed 29 May 2024.

[150] Coalition of Digital, Human Rights and Social Justice Groups, ‘EU’s AI Act fails to set gold standard for human rights’ (Joint Statement/Analysis, 3 April 2024), https://www.amnesty.eu/news/eus-ai-act-fails-to-set-gold-standard-for-human-rights/

[151] Ibid.