Scaling LLM Classification via Hallucination and Vector Search

Classifying data into large taxonomies using LLMs is often inefficient due to token limits and high costs. By prompting a cheap model to 'hallucinate' a plausible category and then using vector search to map it to a real one, developers can achieve better scalability. This hybrid approach combines the creative power of LLMs with the precision of local embedding models.

Key Points

• Traditional LLM classification using large fixed vocabularies is expensive and limited by context window constraints.
• Small, cheap LLMs are capable of generating plausible-sounding categories even without seeing the official taxonomy.
• Hallucinated categories can be mapped to official ones using local vector similarity searches with models like MiniLM.
• This method avoids the need to send massive schemas to the LLM provider for every classification request.

Sentiment

Generally positive and technical, characterized by constructive skepticism regarding accuracy and interest in cost-saving architectural patterns.

In Agreement

• The method is a 'smart' trick for resolving extracted intents to a fixed selection.
• Sending large taxonomies (e.g., 40k tokens) to an LLM is prohibitively expensive and still prone to errors.
• The approach is a significant improvement over having no classification system at all for large datasets.
• It effectively bypasses the token limits and latency issues of traditional structured output methods.

Opposed

• Embedding-based sorts may not be accurate enough for production environments where precision is critical.
• Errors can compound from both the initial LLM hallucination and the subsequent vector similarity search.
• The method might require a 'heavier' model to vet the candidates in the shortlist to ensure reliability.
• A/B testing is necessary to determine if the accuracy loss is acceptable compared to traditional methods.