The data protection lead stopped the fine-tuning workshop ten minutes in. The vendor had brought a "de-identified" customer-support transcript extract: names redacted, account numbers hashed, postcodes and dates of birth still present because "the model needs locality and timing for intent classification." Nobody had written down what an adversary could do with the finished model — only what had been removed from the CSV. That is the gap UK AI teams keep walking into.
Re-identification risk is the probability that someone can link a record — or infer that a person was in a dataset or a model's training run — back to a named individual, using means that are reasonably likely in the circumstances. It is not a nation-state fantasy. It is a linkage problem in tabular releases, and an inference problem once you train on the result.
What UK Law Asks You to Measure
UK GDPR does not reward optimism. Recital 26 and the ICO's March 2025 anonymisation guidance ask whether identification is reasonably likely — taking account of cost, time, available technology, and what auxiliary data exists. You are not required to defeat every theoretical attacker. You are required to stop pretending that theoretical is the same as remote when LinkedIn, the electoral roll, and a public API sit one join away.
Strip direct identifiers and you have finished the first pass, not the assessment. What remains are quasi-identifiers — fields that are not unique alone but become unique in combination, or unique when joined to auxiliary data. British clinicians worked this out in the 1990s: postcode plus date of birth was enough to identify over 99% of UK residents, which is why hospital episode statistics moved to partial postcodes, year-of-birth only, and proper pseudonyms for linkage.
The ICO's motivated intruder test gives you a persona: someone with no inside knowledge, ordinary skill, and access to public resources, who wants to identify a person from your "anonymous" output. If that intruder is likely to succeed, you do not have anonymised data — you have pseudonymised data at best, and personal data in scope.
Linkage Attacks Still Win in Plain Tables
AI does not repeal arithmetic. Before you reach for federated learning slides, the baseline attack still works on the extracts most teams actually fine-tune from.
A Scottish Information Commissioner decision on an NHS out-of-hours dataset shows the shape. Thirty-eight patients in a narrow time window; full postcodes disclosed alongside other fields already held by the requester. The board concluded that in several postcode areas with fewer than twenty residences, identification was a significant risk — not because names were in the file, but because the quasi-identifier combination was.
The same pattern scales to model training. If your "cleaned" CRM or clinical notes retain postcode sector, exact age, gender, and rare event timestamps, you have not moved the data out of identifiability — you have moved it into a GPU cluster. The ONS links administrative datasets on name, date of birth, sex, and postcode under pseudonymisation controls precisely because those fields are how Britain counts people. Your feature store is not magically different because the downstream step is a transformer.
Where AI Changes the Threat Model
Training introduces a second surface. The model can remember, leak, or reveal membership even when the export looked acceptable.
The ICO's AI and data protection guidance names two attacks delivery teams should recognise before they sign off a training run.
Membership inference asks: was this person's data in the training set? Attackers exploit the model being disproportionately confident on records it has seen. Confidence scores and API responses are enough in many setups. The ICO's worked example uses facial recognition: if the model has met this face before, its behaviour betrays that fact even when it reveals nothing else.
Model inversion goes further: given some knowledge of a person already in the training data, an adversary infers additional attributes by probing inputs and outputs. The leak is not a row in a CSV; it is what the weights and predictions encode about that row.
Some architectures make this worse by design. The ICO notes that k-nearest neighbours and support vector machine models can contain parts of the training data in the model itself. Others leak through overfitting: the model memorises specifics instead of learning a general pattern, which is exactly what membership inference feeds on.
Large language models add a third failure mode vendors rarely put on the slide: regurgitation. Training data can reappear in completions. That is re-identification in prose (a support ticket, a clinician's note, a salary row), not in a join table. The European Data Protection Board's 2024 opinion on AI models expects controllers to test against membership inference, model inversion, reconstruction, and regurgitation of training data, and to document the threat model behind those tests.
Why "Synthetic" and "Pseudonymised" Are Easy to Get Wrong
Synthetic data is sold as the escape hatch: generate new rows, ship the generator output, claim you never processed real people in the release. UK law is less impressed by the label than by the residual risk.
If a generative model memorises training records, the synthetic set can still carry personal data, and researchers have shown that vulnerable records in synthetic releases are systematically more exposed to membership inference than aggregate metrics suggest. A board-level "we used a GAN" statement is not evidence; attack results against the specific generator and cohort are.
The PHG Foundation's analysis of synthetic health data under UK GDPR lands in the same place: output data remains personal if it is reasonably likely to be recombined with identifying information, or if identification risk rises as techniques improve. Pseudonymisation with a held key is still personal data in your hands. Anonymisation is a contextual outcome for each release model (public, partner, internal), not a property of the algorithm class you bought.
What the ICO Expects Before You Train or Ship
March 2025 anonymisation guidance is explicit that organisations using data to train AI models should work through identifiability assessment, document it, and review it when new public datasets or attack techniques appear. That is separate from your DPIA, but it should not contradict it.
Before approving a training corpus or a model API, name the release model and the motivated intruder for that model. List quasi-identifiers and rare behavioural fields (query text, ticket IDs, session order), not only demographics. For the ML path, ask whether the architecture retains training examples, how you test for membership inference and inversion, and what happens to erasure requests when personal data may live in the weights.
Prohibit re-identification attempts in contracts. Limit who sees raw extracts. Destroy or return data when the project ends. None of that is cryptography. The ICO treats administrative controls as friction, not proof, but absence of any control is hard to defend when a membership inference paper lands.
The names were redacted before the fine-tuning job ran.
The model still had enough to remember who they were.
What the ICO expects you to document before you train or ship
Name the release model and the motivated intruder
Say whether the extract or API is public, partner, or internal, and describe the motivated intruder for that model. The assessment changes with the audience.
List quasi-identifiers, not only demographics
Include rare behavioural fields — query text, ticket IDs, session order — as well as postcode, age, and gender. Those are how linkage still works after names are gone.
Test the ML path, not only the CSV
Ask whether the architecture retains training examples, and test for membership inference, inversion, reconstruction, and regurgitation before you expose a model API.
Plan erasure when personal data may live in the weights
Document what happens to erasure and rectification requests if the model memorised a record. Retrain or delete may be the honest answer; silence is not.
References
- ICO — How do we ensure anonymisation is effective?
- ICO — About anonymisation guidance (AI training)
- ICO — Security and data minimisation in AI
- ICO — Individual rights in AI systems
- EDPB Opinion 28/2024 — AI models
- IEEE Cipher — UK postcode and date of birth identifiability
- ScotIC — NHS GGC postcode disclosure (2019)
- ONS — Statistical Population Dataset methodology
- PHG Foundation — Are synthetic health data ‘personal data’?
- Giomi et al. — Vulnerable records in synthetic data
Frequently Asked Questions
What is re-identification risk?
Re-identification risk is the probability that someone can link a record — or infer that a person was in a dataset or a model's training run — back to a named individual, using means that are reasonably likely in the circumstances. Under UK GDPR it is a linkage problem in tabular releases, and an inference problem once you train on the result.
Does stripping names anonymise AI training data under UK GDPR?
No. Removing direct identifiers is the first pass, not the assessment. Quasi-identifiers such as postcode and date of birth can still uniquely identify people, especially when joined to auxiliary data. If identification is reasonably likely, you have personal data, not anonymised information.
What is the ICO motivated intruder test?
The ICO's motivated intruder is someone with no inside knowledge, ordinary skill, and access to public resources, who wants to identify a person from your 'anonymous' output. If that intruder is likely to succeed, you do not have anonymised data — you have pseudonymised data at best.
What is membership inference in an AI model?
Membership inference asks whether a person's data was in the training set. Attackers exploit the model being disproportionately confident on records it has seen. Confidence scores and API responses are enough in many setups; the ICO uses facial recognition as the worked example.
What is model inversion?
Model inversion infers additional attributes about someone already in the training data by probing the model's inputs and outputs. The leak is not a row in a CSV; it is what the weights and predictions encode about that row.
Is synthetic data automatically anonymised?
No. If a generator memorises training records, the synthetic set can still carry personal data. UK law looks at residual risk, not the label. A board-level 'we used a GAN' statement is not evidence; attack results against the specific generator and cohort are.