When ‘Seahorse + Emoji’ Hits an Empty Token: Why LLMs Invent the Seahorse Emoji

LLMs confidently believe a seahorse emoji exists and attempt to produce it, but the token doesn’t exist. Logit lens reveals models build a “seahorse + emoji” representation that the lm_head can’t realize, so it snaps to similar emojis like fish or horse. Once the wrong emoji appears in context, some models correct, others loop, and RL likely helps reduce this failure mode.

Key Points

LLMs often hold a latent, incorrect belief that a seahorse emoji exists, paralleling common human misremembering; a seahorse emoji was proposed but rejected by Unicode in 2018.
Logit lens analysis shows models construct a “seahorse + emoji” residual representation in mid-layers, similar to how they form “fish + emoji” for a real emoji.
Because no seahorse emoji token exists, the lm_head snaps this representation to nearby emoji byte vectors (e.g., fish or horse), yielding the wrong emoji.
Autoregressive feedback can reveal the mismatch to the model; some models correct quickly, others loop or spiral.
Reinforcement learning on model rollouts likely helps models internalize lm_head constraints and reduces such failures, unlike base models trained only on next-token prediction.

Sentiment

The overall sentiment of the Hacker News discussion is highly engaged, curious, and largely in agreement with the article's mechanistic explanation of LLM behavior regarding the seahorse emoji. There is a strong undercurrent of humor mixed with serious technical and philosophical inquiry into the nature of AI 'knowledge' and 'thinking.' Many comments express fascination with the phenomenon and a willingness to explore its implications, often by testing various LLMs themselves.

In Agreement

The article's core explanation—that LLMs internally represent the concept of a 'seahorse emoji' but lack a direct output token, leading to the selection of a semantically close but incorrect emoji and subsequent spiraling—was widely affirmed.
Reinforcement Learning from Human Feedback (RLHF) and the integration of external tools (like web search) are seen as crucial mitigations, enabling models to learn from their outputs or access ground truth, thus preventing or resolving the spiraling behavior.
The phenomenon is exacerbated by the fact that humans widely misremember the seahorse emoji's existence (the Mandela Effect), meaning LLMs are trained on data reflecting this collective false memory.
LLMs' self-correction and 'freakout' loops are largely understood as a consequence of their token-by-token generative process, where models react to their own previously sampled (incorrect) outputs rather than having an internal, pre-computation 'thought' process.
The problem extends to other 'plausible but non-existent' emojis or factual inaccuracies, indicating a broader insight into LLM limitations in grounding concepts to verifiable output tokens.

Opposed

Some commenters debated the precise categorization of this behavior, with arguments for it being a 'classic hallucination,' 'confabulation,' or 'probability-based bluffing,' rather than solely a representation/token mismatch.
A technical viewpoint suggested that LLMs are more accurately described as performing 'regularized manifold fitting' rather than being strictly 'statistical or probabilistic' models.
A philosophical perspective argued against anthropomorphizing LLM behavior, emphasizing that 'thinking' or 'freaking out' are metaphors for complex statistical language generation, not genuine cognitive states.
Some view the problem as a fundamental limitation of current LLM architectures (e.g., lack of true internal thinking or hierarchical generation), suggesting that current mitigations like RAG or RLHF are only partial solutions, not addressing the core mechanistic constraint.