What You Need to Know
Bias is a gap in outcomes between groups of people. A protected attribute is a field that records a protected characteristic, such as sex. A proxy is another variable that carries the same information as a protected attribute. A cohort is a group whose outcomes are measured together. A fairness criterion is the rule that decides which gap counts as unfair.
This lesson uses one made-up example throughout: a telecoms provider that uses an AI assistant to decide which applicants for customer service jobs get an interview. It collects no demographic data.
Bias Is Measured in Outcomes
Removing the protected attribute proves nothing, because proxies carry its information. The ICO (the Information Commissioner's Office, the UK data protection regulator) notes that features in the data are often closely correlated with protected characteristics in non-obvious ways. So a model reproduces patterns of discrimination that nobody intended. Modern machine learning makes this outcome more likely, because it finds complex feature combinations and redundant encodings (the same information carried in several variables).
The proxies are ordinary variables: working hours, occupation, application channel, address history and language register (the style and level of formality of a person's writing). In the ICO's example, an employer ranks staff for redundancy (dismissal because their jobs are no longer needed) on hours worked. This discriminates on the basis of sex where more women work part-time.
The telecoms provider's assistant never sees sex, but it sees part-time work history, paper forms handed in at its shops and second-language writing. A design that excludes the sex field and is then declared fair has measured nothing.
Bias also enters through:
- how variables were measured, labelled or aggregated
- the builders' cultural assumptions
- an objective (the goal the model is built to optimise) whose "best outcome" assumes who is worth optimising for
- an interface that is not accessible, which harms a group of people before the model acts
Measure the Outcomes, Then Name Your Criterion
Only cohort-level evidence can show that a fairness claim is true. A strong aggregate accuracy can hide one cohort for which the system performs badly. This happens because the aggregate is a weighted average that the majority dominates. A tighter confidence interval only measures the majority more precisely.
Choose cohorts from a harm hypothesis (a statement of who the system could harm), not from data availability. Ask who this system can disadvantage, and on the basis of which characteristic. Then find the variable you can observe that represents that group, often a behavioural or channel variable rather than a demographic one. The telecoms provider suspects part-time applicants are screened out, so it compares their interview rate with the rest.
Then name the criterion. Three criteria cannot generally all hold at the same time.
- equal error rates across groups: the system is wrong equally often for each group
- equal selection rates: the system selects the same share of each group
- calibration within groups: a given score means the same chance of the outcome in every group
So fairness is a documented choice that is tied to the harm the system can do. It is not a property that a system has or lacks.
The Remediation Ladder
Four fixes follow, cheapest and most durable first.
Fix the data. Rebalance how often each group appears in the data and examine how the labels were produced. Check which population the ground truth (the correct answers the model learns from) came from. A model trained on past decisions learns to repeat them.
Fix the framing. Change what the model is asked to decide, and ask whether that decision is appropriate for a model at all. The ICO asks whether AI using personal data is the best solution in the first place. That question is cheaper to answer before the build.
Constrain the decision. Let the model score and rank rather than decide. Require a human to adjudicate affected cases. Narrow the remit so the contested judgement stays with a person. At the telecoms provider, the assistant ranks and a recruiter decides.
Monitor continuously. Give fairness a live metric, a threshold and an owner, because the population changes over time and a one-off audit becomes out of date. AI decisions also change the world where they are applied. That world produces the next round of input data. Unless someone watches, past unfairness becomes entrenched.
Two actions are not fixes. The first hides the score. That removes the evidence, but the harm remains. The second blames the foundation model's training data and concludes that the application team cannot act. It looks rigorous, but it is wrong. Framing, constraint, remit and monitoring all belong to the application.
Transparency Is Three Obligations
Disclosure to the user, at the point of use. Tell users that they are dealing with an AI system rather than a person. Also tell them the system's limits and how to reach a human. Anthropic's Usage Policy sets the minimum requirement. A consumer-facing chatbot or AI agent that talks directly to external users must tell them that they are dealing with AI. It must do so, at a minimum, at the start of each chat session or in the product interface. Examples of a High-risk AI Recommendation are advice on a legal matter and a decision on a loan. For such a recommendation, the person must also be told that AI was used to produce it.
Explainability to the affected person. The ICO separates process-based explanations from outcome-based explanations. A process-based explanation describes the governance that you followed. An outcome-based explanation gives the reasoning for this decision in plain everyday language. It says whether a human was meaningfully involved, and it gives a timely way to contest the decision. The telecoms provider owes a rejected applicant an outcome-based explanation. A description of the architecture is not an explanation, and "the model determined" is not a basis.
Traceability to the organisation. Record which version, which prompt and which retrieved context produced each output. Retain this record long enough to answer a challenge months later.
Read Which Obligation the Question Is Asking About
A person who asks why their application was rejected wants explainability. A user who does not realise they are talking to a system needs disclosure. A complaint answered months later needs traceability. For each of these cases, there is an option that sounds competent but answers one of the other two obligations.
Accountability Stays With a Person
A named human or role owns the outputs. Automating a decision does not move accountability onto the software. To a regulator or a customer, "the model said so" is not a defence.
Anthropic's commercial terms give the judgement to the customer. The customer decides whether outputs are appropriate before they are used or shared. This judgement includes where human review belongs. Anthropic's own transparency work covers the parts Anthropic builds, not your application.
The ICO expects senior management to approve an AI system's intended use against the organisation's risk appetite (the amount of risk it is willing to accept). The ICO also expects a clear assignment of who is accountable for what from the outset. Where a system affects a person's rights, money, health, safety or opportunities, the design keeps a human accountable and lets the person reach a human. The telecoms provider's head of HR owns the screening outputs.
In the exam
This objective is harder than its share of the exam suggests, because the wrong answers look more rigorous than the right ones. Removing the demographic field sounds careful. Tightening the confidence interval sounds quantitative. Tracing the failure to the foundation model sounds sophisticated. All three leave the affected person in the same position, which makes them attractive under time pressure.
The right answer names a criterion and justifies it against the decision being made. The wrong option asserts that the system is unbiased.
Domain 4 covers how measurement is done. Domain 5 covers the obligation to measure, the threshold at which a gap is unacceptable, and what changes when the number is bad. Domain 3 covers how to build the trace, and Domain 5 covers how long it is kept.
Limits of Fairness Metrics
The ICO says fairness metrics identify discriminatory impacts, but they are not likely to guarantee compliance or explain why the unfairness happens.
Exam Traps
Concluding the system is fair because no protected attribute is collected or used
Proxies correlated with the attribute in non-obvious ways reproduce the pattern anyway, so excluding the field measures nothing. Break outcomes down by cohort against a stated fairness criterion.
Recording the training data of the foundation model as the root cause and documenting the limitation
Base-model fatalism, written to sound like the sophisticated answer. Framing, decision constraint, remit and monitoring are all owned by the application, so an upstream cause presented as an excuse converts a remediable finding into a permanent one.
Offering a model card, a published policy page or a paragraph in the terms as the disclosure obligation
It answers a different question for a different audience. Disclosure is owed to the person at the moment they are using the system. The requirement places it at that moment, and at that moment nobody reads a policy page.
Explaining a decision by describing the model, the pipeline or the architecture
An affected person needs the basis of their own outcome, in everyday language, with a way to contest it. A pipeline description is a process-based explanation offered where an outcome-based one is owed, and it gives them nothing to challenge.
Practice Scenario
A household-insurance assistant triages claims into fast-track settlement or manual investigation. No demographic data is collected. Analysis shows claims submitted through the telephone channel are routed to investigation far more often than identical claims submitted through the app, and telephone claimants are more often older and more often disabled. Investigation adds six weeks. What should the architect do?
Build Exercise
Write the Ethics Page for One System That Decides About People
What you'll learn
- How to write a harm hypothesis and derive cohorts from it rather than from available fields
- How to find the proxies in a feature set and test whether each one carries the protected characteristic’s information
- How to choose a fairness criterion and defend it against the ones you rejected
- How to apply the four remediation steps in order without blaming the base model
- How to turn the three transparency obligations into concrete artefacts with a named accountable owner
- Write the harm hypothesis first: who could this system disadvantage, on the basis of which characteristic, and by how much. Then derive the cohorts you would measure separately from that hypothesis rather than from the fields you happen to hold
Why: Cohorts chosen from data availability measure whatever was convenient. Cohorts chosen from a harm hypothesis measure the thing you are accountable for.
You should see: One or two sentences naming the disadvantaged group and the characteristic, and a cohort list that follows from them.
- List every proxy in your feature set for each protected characteristic in your hypothesis, including behavioural, channel, geographic and language-register variables, and say how you would test whether each one carries the protected characteristic’s information
Why: This is the central claim of the exam objective, in practical form: excluding the attribute proves nothing while the proxies remain.
You should see: A proxy table with a test per row, not a general statement that proxies may exist.
- Name the fairness criterion you have chosen, justify it against the specific harm this system can do, and state what the criteria you rejected would have optimised instead
Why: The criteria conflict, so a claim of fairness with no named criterion is not a claim at all, and the right answer is the option that names one.
You should see: One criterion, one justification tied to the harm, and a short note on each rejected alternative.
- Do the four remediation steps (the rungs) in order, data then framing then decision constraint then monitoring, and record what you would actually change in this system at each step. Do not allow yourself the answer that the base model is responsible
Why: Doing the steps in order puts the cheap durable fixes before the expensive fragile ones, and the refusal is what stops the exercise ending in a documented limitation.
You should see: Four rungs with a specific change on each, including at least one you had not considered before writing it down.
- Write the three transparency obligations as concrete artefacts: the disclosure the user sees at the point of use, the explanation and appeal route an affected person receives in everyday language, and the trace you retain and for how long. Then name the person or role accountable for the outputs and check they can see the basis of what they are accountable for
Why: The three obligations are owed to different audiences at different moments, and accountability is only real when the accountable person can see the evidence.
You should see: Three artefacts written out in the words the reader would actually see, plus a named role and a note on what that role can currently access.
Sources
- What about fairness, bias and discrimination? — Information Commissioner's Office
- How do we ensure fairness in AI? — Information Commissioner's Office
- What is the impact of Article 22 of the UK GDPR on fairness? — Information Commissioner's Office
- Usage Policy — Anthropic
- Commercial Terms of Service — Anthropic
- Transparency Hub — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 5 — Anthropic