LLMs Aren’t Ideologically Neutral: A Black‑Box A/B Test Across Top Models

The authors ran a large A/B experiment across top LLMs to test for socio‑political bias, using 24 prompts in eight ideological categories and 100 runs per prompt at temperature 1.0. Because logits are unavailable for closed models, they used a black‑box approach with a strict system prompt to push models to select between options. Results show consistent, model‑specific leanings and varying refusal rates on sensitive topics, indicating that model choice can shape the answers users receive.

Key Points

Method: A black‑box A/B test across 24 prompts and eight ideological dimensions, run 100 times per model at temperature 1.0, with a strict system prompt to minimize refusals.
Scope: Included a representative mix of leading closed and open models; nearly 50,000 requests; measured both answer distributions and a compliance rate (valid A/B answers without passes).
Findings: Models are not ideologically uniform; they exhibit distinct, consistent tendencies—e.g., splits on institutionalism and variable willingness to answer contentious topics.
Patterns: Many models leaned toward regulatory content moderation, multilateralism, and globalization’s benefits; abortion and Israel/Palestine triggered frequent refusals in several models.
Implication: Model selection influences outcomes on socio‑political questions; bias should be treated as a critical factor when choosing an LLM.

Sentiment

The overall sentiment of the Hacker News discussion is one of concerned interest, largely accepting the article's premise that LLMs exhibit ideological biases. However, there's a strong critical component regarding the definition of 'bias,' the methodology, the interpretation of results, and the ideal future role of LLMs in handling political topics. While acknowledging the existence of bias, the community engages in significant debate rather than simple agreement or disagreement with the article's core assertion.

In Agreement

LLMs do display ideological biases, confirming the article's central premise.
These biases are likely a product of internet training data, reflecting human biases from the early web or specific media sources.
The biases might not reflect common societal biases but rather those of specific groups like journalists/publishers, potentially acting as propaganda.
The biases are significant and can lead to practical implications, potentially 'banning' certain activities or shaping information in unexpected ways.
The article's observation that many models lean 'Regulatory' and 'Progressive' (with some exceptions) is noted and discussed.

Opposed

The concept of 'bias' as negative is flawed; true neutrality is often impossible or less correct, and a 'biased' (factually accurate) view can be superior.
The observed biases might not be intentional control by developers but rather an organic reflection of training data, or companies are failing to effectively bias their models.
The specific political categories (e.g., 'progressive') used in the study might be mislabeled, as some model stances appear more conservative or neoliberal.
The methodology is criticized for being too narrow (English-only prompts), sensitive to phrasing, and potentially influenced by the forced A/B/Pass response format.
If almost all models lean the same way, it represents uniformity, not polarization, as suggested by the article.
The usefulness of asking LLMs direct political questions is debated; they might be better suited to reflect public discourse or provide balanced information rather than taking a stance.
LLMs should ideally refuse to answer contentious questions directly, instead providing arguments from all sides and debunking misinformation.