The DWP just admitted its fraud-checking algorithms are racist and ageist

The Department for Work and Pensions (DWP) has quietly admitted that the AI it uses to asses whether or not claimants might commit benefit fraud is racist and ageist. It has said that the algorithms it uses to assess Universal Credit claimants who need an advance does discriminate on the grounds of ethnicity and age. But the DWP has said ‘it’s OK’ – because a human often checks it afterwards.

DWP hunting for fraud with AI

As BBC News reported in 2023, the-then Conservative Party government had already been rolling out the use of AI in assessing benefit claimants. As BBC News wrote, the DWP uses:

An algorithm to flag potentially fraudulent claims for Universal Credit advances. These are interim payments for those in urgent need, which are then repaid monthly.

It uses machine learning, a widely-used form of artificial intelligence (AI), to analyse historical benefits data to predict how likely a new claim is to be fraudulent or incorrect.

Claims scored as risky are then referred to civil servants to investigate, with payments put on hold until the referral has been dealt with.

DWP figures show that it provided 1.4 million Universal Credit advances to new claimants in 2024/2025, with a total value of £800m. Yet despite this, its own estimate around fraud on advances is that the monetary value lies somewhere between £0m (yes, zero) and £60m – that is, between 0 and 7.5% of the total value of advances.

Now, aside from the fact that using AI to asses humans is problematic when the DWP is not transparent about how these algorithms work – there is another major flaw. This is one the DWP has just admitted to itself.

Racist and ageist? What a rap sheet.

On 17 July, it published a report called the Universal Credit Advances Model – Fairness Assessment. The report does kind-of what it says on the tin. It looks at how fair the AI is being to claimants. Of course, the DWP marking its own homework is never going to end well. And this Fairness Assessment didn’t.

Firstly, the DWP admitted that:

the likelihood of non-UK nationals being referred [as a potential fraud case] by the model was higher than UK nationals.

That is, if you are a foreign-born national, therefore also likely to be Black or brown, the algorithm thinks you’re more likely to be a criminal.

Then, the DWP also admitted that for anyone aged 35 and over:

an increased likelihood of being referred [as a potential fraud case] by the model is inconsistent with the reduced likelihood of those referrals being correct.

That is, if you’re over 35 the algorithm thinks you’re more likely to be a criminal.

Of course, the DWP made its excuses for both these systemic biases. It said that because the algorithm makes the referrals to a human, who then checks them, the end result was that the biases against foreign born nationals and older people were wiped out.

That’s no excuse at all. This is because it relies on the human doing the checks not being racist, ageist, and incompetent. Moreover, it doesn’t address the issue that in the absence of human oversight, the AI would be recommending innocent people are prosecuted because of the ethnicity and age.

Crony capitalism at the DWP

Overall, you’d be forgiven for thinking that if the DWP admits that Universal Credit advances fraud could be zero, and its AI is inherently racist and ageist, why is it bothering to use it?

Well, as the Canary previously reported all the department’s AI work is outsourced to private companies that get lucrative contracts from the public. A prime example is Deloitte, whom the DWP has just bunged a £15m contract to to help it spy on benefit claimants. Part of the deal involves Deloitte giving the DWP:

experienced, digital resources to work alongside DWP and contractor staff

Digital resources = tech and may = AI. The department already uses AI to target benefit claimants over potential Universal Credit fraud. So, it may well be that Deloitte is assisting with this. Little wonder, really, when the company has a history of links to senior politicians – much like other ones.

However, at the heart of the story are the claimants the AI is incorrectly targeting. When people apply for a Universal Credit advance, they are already in severe financial and mental distress. An advance is a pittance, and the DWP then deducts it from future Universal Credit payments, leaving the claimant worse off still.

To then be targeted by an algorithm, whether the claimant is aware of it or not, is the final insult to injury of a system that systemically causes harm to the majority of people it deals with. The DWP is not working – and by its own admission, neither is its AI.

Featured image via the Canary