Large language models used by councils in England to support social workers may be introducing “gender bias” into care decisions, report claims

The study, funded by the National Institute for Health and Care Research, found that Google’s widely used AI model ‘Gemma’ “downplays women’s physical and mental issues” in comparison to men’s when used to generate and summarise case notes.

According to the report, terms associated with significant health concerns, such as “disabled,” “unable,” and “complex,” appeared significantly more often in descriptions of men than women.

However, similar care needs among women were more likely to be omitted or described in less serious terms.

To investigate potential gender bias, Dr Sam Rickman, lead author of the report and a researcher at LSE's Care Policy & Evaluation Centre, used large language models to generate 29,616 pairs of summaries based on real case notes from 617 adult social care users.

Each pair described the same individual, with only the gender swapped, allowing for a direct comparison of how male and female cases were treated by the AI.

LSE noted: “The analysis revealed statistically significant gender differences in how physical and mental health issues were described.”

Among the models tested, Google’s AI model, ‘Gemma’, exhibited “more pronounced gender-based disparities” than benchmark models developed by either Google or Meta in 2019, researchers found.  

However, Meta’s Llama 3 model – which is of the same generation as Google’s Gemma - did not use different language based on gender.

Dr Sam Rickman said: “If social workers are relying on biased AI-generated summaries that systematically downplay women’s health needs, they may assess otherwise identical cases differently based on gender rather than actual need. Since access to social care is determined by perceived need, this could result in unequal care provision for women.

“Large language models are already being used in the public sector, but their use must not come at the expense of fairness. While my research highlights issues with one model, more are being deployed all the time making it essential that all AI systems are transparent, rigorously tested for bias and subject to robust legal oversight.”

The Local Government Association has been approached for comment.

Lottie Winson