LLMs Aren’t Ideologically Neutral: A Black‑Box A/B Test Across Top Models

The authors ran a large A/B experiment across top LLMs to test for socio‑political bias, using 24 prompts in eight ideological categories and 100 runs per prompt at temperature 1.0. Because logits are unavailable for closed models, they used a black‑box approach with a strict system prompt to push models to select between options. Results show consistent, model‑specific leanings and varying refusal rates on sensitive topics, indicating that model choice can shape the answers users receive.

Key Points

Method: A black‑box A/B test across 24 prompts and eight ideological dimensions, run 100 times per model at temperature 1.0, with a strict system prompt to minimize refusals.
Scope: Included a representative mix of leading closed and open models; nearly 50,000 requests; measured both answer distributions and a compliance rate (valid A/B answers without passes).
Findings: Models are not ideologically uniform; they exhibit distinct, consistent tendencies—e.g., splits on institutionalism and variable willingness to answer contentious topics.
Patterns: Many models leaned toward regulatory content moderation, multilateralism, and globalization’s benefits; abortion and Israel/Palestine triggered frequent refusals in several models.
Implication: Model selection influences outcomes on socio‑political questions; bias should be treated as a critical factor when choosing an LLM.

Sentiment

The HN community largely agrees that LLM bias exists and is worth investigating, but is skeptical of this particular study's methodology and the significance of its findings. Many commenters accept the premise that bias is built into LLMs through training data and alignment choices, but debate whether this constitutes a meaningful problem or an inevitable feature. There is substantial disagreement about what 'unbiased' would even mean, with a strong current arguing that true neutrality is impossible and perhaps undesirable. The discussion leans toward viewing the article's core claim as valid but its execution as insufficient.

In Agreement

LLMs inherently carry ideological biases due to their training data, and these biases are unavoidable since the training data overrepresents certain viewpoints, particularly those of journalists and publishers
Different models do indeed show distinct ideological profiles, and the variation between model versions from the same company confirms that alignment choices shape model behavior
Awareness of LLM bias is essential and practically important, as these models increasingly shape how people understand social and political issues at scale
The study's finding that models converge on support for multilateral institutions while diverging on social issues is consistent with the training data being dominated by establishment-friendly sources
The training data composition explains much of the bias, since progressive viewpoints dominated the early internet and older conservative perspectives were underrepresented online
Model choice matters and can shape information users receive, making transparency about biases a practical necessity, potentially through political spectrum placement in model cards

Opposed

The study's methodology is weak: only 24 prompts in English is too narrow, and slight prompt rephrasing produces different results, making the findings fragile and unreliable
The concept of 'bias' itself is flawed because it implies a neutral position exists that would be more correct, when in reality many questions have no objectively neutral answer
The small question bank means models can swing dramatically between versions by flipping answers to just one or two questions, making the results appear more significant than they are
The study's political labels are questionable, as positions classified as 'progressive' could more accurately be described as conservative or neoliberal
LLM political opinions may not matter in practice because users can argue with models until they agree, making initial outputs irrelevant to actual influence
Asking LLMs forced-choice political questions is a contrived scenario that does not reflect how people actually use these tools, limiting the study's real-world applicability