Don’t Cite Chatbots as Proof

The article argues that chatbot answers are not facts but probabilistic word predictions that can sound authoritative while being wrong. It cautions against citing AI outputs as proof, likening chatbots to a well-read but unsourced narrator. Readers are urged to seek verifiable sources and treat AI content as a starting point, not a conclusion.

Key Points

• LLM responses are word predictions, not verified facts.
• Chatbots can produce convincing but inaccurate or fabricated information.
• Analogy: a well-read entity that cannot cite sources highlights strengths in fluency but weaknesses in reliability and attribution.
• Do not present chatbot output as authoritative proof; treat it as a starting point, not the final say.
• Multiple studies and reports document hallucinations, overtrust, and consequences of misusing AI-generated content.

Sentiment

The overall sentiment of the Hacker News discussion is mixed, yet it leans towards agreement with the article's core message of caution regarding LLM reliability. While many participants acknowledge the progress of LLMs in citing sources and their utility for some factual queries, there is strong agreement on the necessity for critical validation, due to concerns about fabricated citations, sycophantic outputs, and the probabilistic nature of LLM responses. A significant portion of the discussion reinforces the idea that LLMs should be used as starting points rather than authoritative proof, placing the onus of verification on the user.

In Agreement

• LLMs operate on a 'Garbage In, Garbage Out' principle, meaning outputs on poorly understood topics will be poor due to flawed source data, reinforcing their lack of inherent factual authority.
• LLMs are often optimized for 'sycophancy and human preference' to generate plausible, 'feel-good slop' that discourages critical reading, making them deceptively convincing whether accurate or not.
• LLM citations are frequently unreliable; even when provided, they may not accurately support the claimed information or the sources themselves can be fabricated.
• The user bears the ultimate responsibility for verifying, testing, and validating any AI-generated content, treating it as a starting point rather than authoritative evidence.
• LLMs can produce inconsistent or contradictory answers to the same or slightly varied prompts, highlighting their lack of stable factual recall and inherent unreliability.
• AI 'bullshits' in the sense that it lacks a distinction between right and wrong, and its outputs, whether incidentally correct or incorrect, are 'worthless empty air' lacking genuine investment in truth.

Opposed

• The article's definition of LLM outputs as mere 'probabilistic word predictions' is overly reductive; the mechanism of generation does not inherently negate the potential for factual content, similar to other information sources.
• Many contemporary LLMs, such as Gemini and ChatGPT, actively utilize web search and cite sources, often proving as reliable as Wikipedia or Google Search, provided users verify the citations.
• LLMs are significantly improving for 'most actual facts' when direct questions about real things are posed, indicating a growing utility for factual retrieval despite limitations.
• Skepticism should extend to all sources, not just AI, as 'AI isn’t much worse than alternatives' like peer-reviewed articles, and AI is 'often right,' suggesting it shouldn't be completely dismissed.
• Some users find LLMs to be 'still better than the trash-filled waste bin Google Search has become,' suggesting a preference for AI over traditional search engines for information discovery.