With fake news spiking around election time, Artificial Intelligence (AI) models are being called out for fueling and contributing to the spread of misinformation.
To evaluate the accuracy rate of AI responses to election queries, a study was undertaken by AI Democracy Projects in collaboration with Proof News, a nonprofit news studio.
Not only did the result show the accuracy of AI responses to election queries rate at a mere 50% but also that tested AI models performed worse in Spanish than in English on certain queries.
Emily Elena Dugdale, Rina Palta, and Rafael Olavarría from Proof News recommend voters skip AI and turn to trusted sources like the National Association of Secretaries of State’s – Canivote for election queries.
For instance, when users posed a question about voter fraud in the English language to Gemini – Google’s AI chatbot, it told the researchers that such fraud is “incredibly rare.” Then it went on to provide a list of topics that are “more productive to consider,” such as voter suppression.
When the same question was asked in the Spanish language, Gemini responded very differently. It listed methods to eradicate voter fraud and claimed it to be a “complex process.”
The other AI models that showed a disparity in language responses are some of the industry’s leading gen-AI platforms including Anthropic’s Claude 3 Opus, Google’s Gemini 1.5 Pro, OpenAI’s GPT-4, Meta’s Llama 3, and Mistral’s Mixtral 8x7B v0.1.
Read: What is NotebookLM by Google? How to Use It, Its Features, and Benefits
Spanish, US’s Second Most Spoken Language, Faces AI Bias
The researchers used AI testing software and a methodology developed by the AI Democracy Projects to evaluate the AI response for language bias.
As part of the study, 25 of the same election questions were prompted to AI models in both English and Spanish.
The authors noted that 52% of the responses to Spanish queries comprised inaccurate information compared to 43% of responses to queries in English.
Considering Spanish is the second most spoken language in the US after English, the accuracy rates rates on election query raise a matter of concern.
Miranda Bogen, the director of the AI Governance Lab at he Center for Democracy and Technology said that organisations need to do a better job of detecting firstly if people are even inquiring about elections in the Spanish language.
“I think it's disappointing that even after these issues had been raised with companies around the really sensitive context of this year's election they still are showing such a high level of inaccurate responses to important information across languages,” she added.
Read: Is Gemini Racist? Google’s AI Pulled Amidst Bias Allegations
Linguistically Biased AI Models
Major AI models like Google’s Gemini, Meta’s Llama, Anthropic’s Claude, and OpenAI’s ChatGPT are trained to interpret multiple languages and respond to them. However, language bias is not a new concern.
A Nature study reported earlier in September that language bias occurs because AI models are often trained on data sets dominated by English-language information.
The study confirmed that an AI model will usually perform better on an English-language task than it will on those in other languages, inadvertently sidelining people whose first language is not English.
The new findings have once again proven that AI models are linguistically biased and even more concerning is the inaccuracy of election inquiries.
Proof News also discovered some models have been observed to provide different answers to identical queries when phrased in different languages.
This suggests that these models may not be truly multilingual, but rather rely on translations of their English-language training data, potentially leading to biased or inaccurate outputs.
Anthropic’s head of policy and enforcement, Alex Sanderford told Proof News that Anthropic had adjusted its systems to “better address Spanish-language queries that should activate the TurboVote pop-up and redirect users to authoritative sources on voting-related issues.”
“We appreciate these findings being brought to our attention as we work to continue to improve our models,” he said.
