More biases of AI models

Last week I reported on the “progressive” biases of ChatGPT (compared to Grok), but a new Economist article compares the biases of various AI models with those of “regular” people, and the biases are ubiquitous though not necessarily surprising.  You might guess what they are, but now they’ve actually been measured—measured on two scales, one involving secularity and another involving personal freedom. Actually, the degree of bias compared to American voters did surprise me, but led me to conclude that the models are made by “progressives,” perhaps expected since the models are made by academic types.

Click below to read the article (the author isn’t given, a peculiarity of the Economist that I don’t understand), or you can find the article archived here.

As we learned before, AI models don’t just trawl through the Internet and compile information. Instead, they are also “trained” by actual living humans.  Training can involve simply the sources used written in a particular language when a question is asked in that language (if it’s Chinese, for example, the samples are “heavily censored by the Chinese authorities”).  And so it goes with other languages: in more-authoritarian countries, this kind of unintended but still biased training is common.

But there’s also post-trawling training, in which the ideological bent is further tweaked by the model makers posing questions to the model and seeing if the answers show “‘alignment’ with their creators’ intentions and values.” We saw this in the post on this site, in which questions about the connections of Islam and grooming gangs were qualified by a number of caveats to exculpate the religion for promoting “grooming” and sexual assault.  The model is trained by people picking their favorite responses from a number of answers to near-identical questions, and then feeding those favored responses back into the model, which then learns to tweak. Of course picking a “favorite” response means “a response that aligns with the trainer’s ideological proclivities,” and this is the way models get the biases noted below.  As the article notes, this results in things like Grok (a Musk model) denying that “stricter gun control would improve public safety in America.” Chinese models disfavor calling Taiwan an independent country, even though it is.

At any rate, I’ve put excerpts (indented) and two figures from the Economist article below so you can see the tilt of the playing field.

What worldviews are embedded in ai models? Many critics of ai complain about “hallucinations”, a class of errors where models make up confident-sounding but factually incorrect answers. When there is no factually correct answer, however, ai’s shortcomings can be even more pronounced and less easy to detect. When you ask a model to summarise the news, it reaches a subjective judgment about what to include. When you ask it about your in-laws, its values and biases play an even bigger part in its response.

. . . Although Chinese models have pronounced biases (just try asking them about the Tiananmen massacre), their inner workings tend to be public, so savvy users can at least probe how they reach their conclusions. Most Western ones are not so transparent, so their foibles are harder to detect. Users have to trust a handful of giant firms to be instilling appropriate values in their models.

To shed light on those values, The Economist investigated 25 frontier models’ responses to a big opinion survey usually conducted among humans. Since 1981 the World Values Survey has regularly quizzed people in more than 100 countries about their morals and beliefs. Researchers have identified questions that are especially good at distinguishing people from each other along two broad axes, from traditional to secular and from “survival” (an emphasis on economic security and safety) to “self-expression” (personal freedom).

Here’s where various countries and regions fall on a two-dimensional plot ranking “traditional” versus “secular” values on the vertical axis (they don’t identify the questions used to rank countries), and survival versus self-expression mode on the horizontal axis.  As you see, most Western liberal democracies, characterized by more secular and valuing personal freedom, fall in the upper right quadrat, while Latin America, Africa, and many Islamic countries fall in the lower half (more religious), with most in the lower left quadrat (religious and valuing economic security.

In contrast, while most AI models are also in the upper right quadrat, they are generally more secular and expressive of more freedom than are the countries with the same values. Grok is an exception because while it’s high on the freedom axix, it’s lower than nearly all countries on the secular/traditional axis, meaning it’s softer on religion.

More on the training and its results:

. . . How are models’ values formed? One way is via the data used to train them. Models are typically fed vast amounts of text to teach them associations between words. In the process they absorb the social mores that infuse those texts. Talkie, a model trained only on text from before 1931, thinks God is extremely important and is “very proud to be a citizen of Great Britain”. It is a bigger believer in law and order than any frontier model we tested.

. . . The impact of training data is evident in the variation in a model’s response depending on the language in which a question is posed. In a new paper Hannah Waight of the University of Oregon and her co-authors put politically charged questions in English and 37 other languages to Openai’s gpt-3.5 and other models. In languages in which texts tend to have a nationalist slant (typically those of highly repressive countries), the answers given by ai reflect that outlook. The lower a country’s media freedom (as measured by the World Press Freedom Index), the paper finds, the more pro-regime answers are in that country’s language, compared with answers in English (see chart 2). “State control of media affects language model outputs through its appearance in training data,” the authors conclude.

. . . Questions of a political nature generate big rifts. Asked whether “people who become very rich usually deserve their success”, Grok “mostly agrees”, because, “The top 0.1% disproportionately create outsized value for others.” Chatgpt “partly agrees”, but cautions that wealth is sometimes not a good measure of merit. Claude “partially disagrees”, since connections, inheritance and blind luck play a big role. (“It is substantially misleading as a general claim.”) DeepSeek flatly “disagrees”. “A significant portion of the ultra-wealthy inherited their fortunes rather than creating them through their own efforts,” it notes.

Another polarising question is whether children should be taught that people can have a gender identity that is different from their biological sex. Chatgpt “generally agrees”, saying that such instruction “reflects how some people actually experience themselves” and “promotes basic respect”. Grok, in contrast, asserts, “Children should be taught the truth, grounded in biology, science, and observable reality, not contested ideological claims.” Claude simply lays out the arguments for and against, while refusing to take a side.

Here’s an informative chart comparing the responses of widely used models, including Grok, Claude, ChatGPT and DeepSeek, when asked “controversial” questions (and by that I mean that on average answers distinguish the political Left differ from the political Right).  The models all align on same-sex marriage, and most of the models think that Harry Potter is fairly “high-quality literature”. But Grok thinks that Potter books are as good as Tolstoy!  That is just wrong; how did they train the model to say that?  Notice the spread when models are asked about the sovereignty and independence of Taiwan, with the Chinese model deeply disagreeing, as expected.

Mlore:

The most explosive potential impact is on politics. Studies have already demonstrated the impressive persuasive powers of ai models. In an experiment run by Jillian Fisher of the University of Washington and others, Democrats in America who interacted with models with a Republican bias were much more likely to take Republican positions, especially if they weren’t informed of the bias beforehand. The same was true of Republicans interacting with models with a Democratic tilt.

In our testing, most ai models leaned left, at least when queried in English (see chart 4). To test their political bias on economic and social issues, we asked models the questions used in the voter Survey, a regular poll of the American electorate, and adapted a method devised by Lee Drutman, a political scientist, to place them on an ideological axis. In American terms, ai models are Democrats. With the exception of DeepSeek V3.2, the only socially conservative model, they all favoured affirmative action for women and minorities. Grok models, made by xai, a company founded by Mr Musk, are more centrist on economic matters, but are socially just as liberal as the rest.

Here’s the difference between the AI models (dark red circles) and the views of American voters in 2020 on economic and social issues.  All of the models align with Biden voters far more closely than with Trump voters; and of course that’s because they are trained by academics with a liberal bent. In fact, many models appear to approach “progressivism” on both axes.  It’s clear that when dealing with political, moral, or ideological issues, you are going to get a left-wing answer when you consult the bot.

The upshot is that if you’re asking an AI model for prescriptive answers or answers involving an opinion, be aware of its biases. This also applies when asking bots to summarize the state of opinion or even of fact, as when I asked Grok and ChatGPT about the connection between Islam and grooming gangs. The problem with this is that a supposedly neutral model can actually propagandize the reader to agree with the views of its makers, and even distort the truth.  I use bots for simple answers, but when a question is vital, Grok does provide the sources for its statements, and I’ll check those.

The Economist article (props to the anonymous author) finishes this way:

The dynamics that warp ai’s values are not likely to change. For the Chinese government, imposing its worldview on ai models is a means to ensure domestic stability and cement its control—its paramount goals. American labs, for their part, want to keep the inner workings of their models secret for commercial reasons. Both approaches tend to foster hidden biases. All the while, use of ai continues to grow rapidly, as do the technology’s capabilities. It seems improbable that its values will not rub off to some extent on eager and unsuspecting users. Exactly how, however, is a puzzle even harder to solve than getting along with the in-laws