Scaling LLM Classification via Hallucination and Vector Search

Classifying data into large taxonomies using LLMs is often inefficient due to token limits and high costs. By prompting a cheap model to 'hallucinate' a plausible category and then using vector search to map it to a real one, developers can achieve better scalability. This hybrid approach combines the creative power of LLMs with the precision of local embedding models.

Key Points

• Traditional LLM classification using large fixed vocabularies is expensive and limited by context window constraints.
• Small, cheap LLMs are capable of generating plausible-sounding categories even without seeing the official taxonomy.
• Hallucinated categories can be mapped to official ones using local vector similarity searches with models like MiniLM.
• This method avoids the need to send massive schemas to the LLM provider for every classification request.

Sentiment

The overall sentiment is cautiously interested but skeptical. Hacker News broadly recognizes the method as a clever hybrid of language generation and vector retrieval, especially for large taxonomies, but many commenters want stronger comparisons against simpler baselines and clearer framing of the generated label as an intermediate semantic representation rather than a final answer.

In Agreement

• The approach is a practical way to use an LLM for semantic query expansion while leaving final classification constrained by vector similarity against known categories.
• Generated intermediate labels can capture user intent better than the raw input when the taxonomy wording is large, awkward, or mismatched with natural descriptions.
• The pattern resembles useful retrieval practices such as embedding summaries, semantic routing, or generating representative text before matching against a high-cardinality set.
• Using a cheap model for the semantic bridge and a fast vector index for grounding can reduce the need to send a large taxonomy through the model context.

Opposed

• Calling the intermediate output hallucination makes the method sound like intentionally asking for wrong answers and then hoping a later mapping step fixes them.
• It is unclear whether the LLM-generated label meaningfully outperforms direct embedding similarity between the input and the category labels.
• The extra model call may add latency and cost compared with search, database queries, masked logits, or better indexed category metadata.
• The technique still depends on embedding quality and may not guarantee consistent classifications when the generated proxy varies.